Nx Witness Integration

This documentation is intended to provide you an overview and the installation details of the integration between Nx Witness VMS and SmartFace Platform.

The integration between Nx Witness VMS and SmartFace Platform enhances the Nx Witness with face recognition capabilities. The main advantage of this integration is that information processed by SmartFace Platform is displayed and stored by Nx Witness VMS together with processed video streams.

Key features overview

Nx Witness displays and stores the following information from SmartFace:

  • Face detection

  • Face tracking

  • Age and gender estimation

  • Face mask status

  • Identification of a person

  • Information about a person not identified against any of the watchlists

  • Person identification spoofing

  • Cropped faces

The information in Nx Witness is provided in the form of objects and analytic events.

Architecture overview

Integration between the SmartFace facial recognition platform and Nx Witness VMS is performed through the SmartFace Analytics plugin and SmartFace Camera services installed on Nx Witness server.

SmartFace Analytics plugin running within Nx Witness is responsible for the communication and transformation of data received from SmartFace. SmartFace Camera services installed on Nx Witness server sends processed frames into SmartFace Platform. The Camera services perform also lightweight processing in form of tracking of detected faces. SmartFace Platform detects faces on incoming frames, extracts demographic data from the detected faces and matches the faces against watchlists stored in SmartFace. All information obtained by SmartFace is sent back to Nx Witness. SmartFace Analytics plugin then processes data coming from SmartFace Platform, so that Nx Witness can display it.

As the integration is backed by SmartFace Platform all SmartFace features are available to you.

⚠️ The configuration of these features is performed in SmartFace.

Installation

For the proper functioning of the integration, it is required to install SmartFace Platform, Nx Witness VMS and the SmartFace analytics plugin. SmartFace Platform and Nx Witness VMS should be successfully installed and configured before you start to install the integration components.

The Nx Witness VMS can be installed on the same machine as the SmartFace Platform, or on a separate machine. We recommend installing SmartFace Platform and Nx Witness server on separate machines. Please consider which option you choose based on your hardware infrastructure.

Instructions which describe both options can be found in these chapters:

⚠️ SmartFace Platform and Nx Witness VMS are resource intensive, therefore installing full SmartFace Platform alongside Nx Witness VMS on the same machine is not recommended. We recommend running only the SmartFace Camera services on the same machine as Nx Witness server. SmartFace Camera services are lightweight and they process raw frames coming from Nx Witness server which produces raw frame data. Having SmartFace Camera on same machine as Nx Witness would ensure the data isn’t sent across the network, potentially limiting the throughput. A single Full HD raw frame is ~6.2 MB, so at 30 FPS one camera processing takes ~186 MB/s of network traffic, which may cause network contention and slow down the video processing performed by SmartFace.

To install and configure SmartFace integration with Nx Witness on a single machine, follow these steps:

Install SmartFace Platform

Install the SmartFace Platform. For information on how to install SmartFace Platform, see Installation on Windows or Installation on Linux.

⚠️ In the step of Camera setup configuration, you should choose the number of slots according to the number of Nx cameras which need to be processed.

After a successful SmartFace Platform installation, ensure that MS SQL Server and RabbitMQ are reachable from the machine on which Nx Witness is hosted:

Install Nx Witness VMS

Install the Nx Witness VMS v5.X or later. You will need both Server and Client, as the Client will be used in configuration. Client should be typically installed on VMS operator’s machine.

Install and configure the integration components

  1. Stop the Nx Witness server

    • On Windows by right-clicking the tray icon and selecting Stop Server.
    • On Linux (Ubuntu) with sudo service networkoptix-mediaserver stop
  2. Download the SmartFace analytics plugin from our Customer Portal and unzip.

    • On Windows copy the .dll file to {NxInstallPath}\plugins\ usually on C:\Program Files\Network Optix\Nx Witness\MediaServer\plugins. See bellow:

    - On Linux - create folder `smartface_analytics_plugin` inside the `{NxInstallPath}/bin/plugins` usually on `/opt/networkoptix/mediaserver/bin/plugins` - copy `.dll` into this folder. See bellow:

  3. (optional) In case you would like to brand the plugin with your own labels, specify the plugin white labeling, i:

    • Create sf_plugin_branding.json file in the installation path of Nx Witness server

      • On Windows usually in C:\Program Files\Network Optix\Nx Witness\MediaServer\
      • On Linux in /opt/networkoptix/mediaserver/
    • Use formatting of sf_plugin_branding.json file below:

    {
        "product_name": "SmartFace Platform",
        "plugin_name": "SmartFace analytics plugin",
        "vendor_name": "Innovatrics"
    }
    
    • Change labels for your desired branding
    📢 It is important to create the branding file with branding changes prior to the first start of Nx Witness server with SmartFace analytics plugin. Otherwise you won't be able to change the branding of the plugin anymore. When plugin is loaded for the first time Nx Witness caches a plugin information about the branding into the database which can not be modified later on without an uninstallation of Nx Witness!
  4. Start the Nx Witness VMS.

    • On Windows by right-clicking the tray icon and selecting Start Server.
    • On Linux (Ubuntu) with sudo service networkoptix-mediaserver start
  5. In the left panel, right-click the camera you want to process with SmartFace and select Camera Settings. In the Plugins tab, enable the SmartFace analytics plugin .

    Repeat the step for each camera that you want to process with SmartFace.

  6. In the Camera Settings of the analytics plugin, set a unique port for each camera that should be processed by SmartFace. We recommend to start from port 4000. This is a TCP port which is used for a communication between the Nx Witness plugin and SmartFace Camera services.

    • On Windows by right-clicking the tray icon and selecting Start Server.

    • On Linux (Ubuntu) use named hostname. In this sample we deploy SmartFace alongside NX Witnes on host named nx-1u

  7. Download and unzip current SmartFace_Binaries_Package_v5_X.YY.zip from our Customer Portal

    1. Configure the SmartFace Camera service with the MS SQL Server connection string and RabbitMQ address and credentials. This is performed by modifying the SmartFace.appsettings.json file:
    • Configure the MS SQL connection string under the key ConnectionStrings.CoreDbContext to the value you have set during the SmartFace installation.

    • Configure the RabbitMQ under the keys RabbitMQ.Username, RabbitMQ.Password, RabbitMQ.VirtualHost, RabbitMQ.Hostname and RabbitMQ.Port to the values you have set during the SmartFace installation.

    1. To start the SmartFace Camera service for each Nx Witness camera that you want to process with SmartFace:
    • Run the Service_Installer.bat script as an administrator.

    • When prompted to enter service you would like to install, insert camera.

    • For the service name of the SmartFace Camera Service on Nx Witness server, enter the Nx ID (UUID) of the Nx Witness camera that you want to process. To obtain the Nx ID: Go to Camera Setting (right-click on a particular camera) and under the General tab, select More info. Then, copy the Nx ID from the Camera ID field.

  8. Create a camera using the SmartFace API, hosted on the server where SmartFace Platform is installed

    • SmartFace Platform is managed through the API. By default, the API is hosted on port 8098 and Swagger UI is available on localhost:8098/index.html

    • The endpoint to create the camera is POST /api/v1/Cameras. To find it in Swagger UI, go to the Camera section, open the POST call and click Try it out. Delete the prefilled request body and then fill it according to the following Example request body:

        {
            "serviceName": "338ef68-e0e6-c775-9459b244c102",
            "name": "My camera",
            "source": "sfcam://localhost:4000",
            "enabled": true,
            "faceDetectorResourceId": "cpu_remote",
        }
    
    • Set the serviceName property to the Nx camera ID. The instructions on how to obtain the Nx ID are described above. This step binds the SmartFace Camera service with the camera created on the API.

    • Set the name property as an arbitrary string used to identify the camera.

    • Set the source property to the custom string in the format sfcam://<host>:<port>. For example, sfcam://localhost:4000 where:

      <host> is the address (IPv4 or hostname) of the machine where Nx Witness is hosted. As the Nx Witness and Camera services should be hosted on the same machine, localhost should suffice.

      <port> is the TCP communication port which you have set up in the Nx Witness camera plugin settings.

    • Preferably, set the remote detection for the SmartFace Camera service as follows: set the FaceDetectorResourceId to either cpu_remote or gpu_remote, depending on whether you want to use GPU.

    • (Optional) Configure other camera parameters depending on your requirements.

    • To send the request, click Execute.

    When the SmartFace camera is enabled, then the video processing should be running and face objects should be available in Nx Witness.

  9. Create a camera using the SmartFace API:

    • SmartFace Platform is managed through the API. By default, the API is hosted on port 8098 and Swagger UI is available on localhost:8098/index.html.

    • The endpoint to create the camera is POST /api/v1/Cameras. To find it in Swagger UI, go to the Camera section, open the POST call and click Try it out. Delete the prefilled request body and then fill it according to the following Example request body:

        {
            "name": "My camera",
            "source": "sfcam://localhost:4000",
            "enabled": true,
        }
    
    • Set the name property as an arbitrary string used to identify the camera.

    • Set the source property to the custom string in the format sfcam://<host>:<port> . For example, sfcam://localhost:4000 where:

    • <host> is the address (IPv4 or hostname) of the machine where Nx server is hosted as accessible from the SmartFace Camera service. In most cases the value localhost should suffice.

    • <port> is the TCP communication port that you have set in the Nx Witness camera plugin settings.

    • (Optional) Configure other camera parameters depending on your requirements.

    • To send the request, click Execute .

When the SmartFace camera is enabled, then the video processing should be running and face objects should be available in Nx Witness.

SmartFace data displayed in Nx Witness

In this chapter you can find information about the SmartFace information displayed in Nx Witness and also about how to configure this information.

Faces which are detected by SmartFace are displayed as Face objects in the Nx Witness VMS video items. The position of the face is tracked as it moves in the video, which enables Nx Witness to render bounding boxes around the detected faces.

All detected faces are automatically matched against all members saved in watchlists available in SmartFace. If the detected face is matched against a watchlist member in SmartFace, then information about the watchlist name, watchlist member and the matching score is also displayed.

For an identified person Nx Witness displays the following attributes:

  • Age – Estimated age

  • Gender – Female or Male

  • Face mask – Information if the detected person is wearing a mask

  • Name – The name of the person who is saved as a watchlist member

  • Id – ID of the watchlist member

  • Watchlist – The name of the watchlist in which the detected person is stored as a watchlist member

  • Score – The matching score of the detected face and the matched watchlist member

  • IsSpoof - True or False (if spoof detection is configured).

Face preview in the Objects tab.

To display the face attributes, select the Objects tab in the right pane. After that the face attributes are displayed in two locations – on the camera item, in the Overlay info and in the right panel, in the Objects tab. The difference between these views is that the Overlay info displays face attributes continuously from the obtained information, but the Object tab displays the first obtained face attributes and marks other information in brackets.

Configure which SmartFace information is displayed

  1. In the left panel, right-click the camera for which you want to set up which SmartFace information is displayed and select Camera Settings.

  2. Go to Plugins and select SmartFace analytics plugin.

  3. To enable or disable a face tracking attribute, mark or clear the relevant checkboxes in the Object tracking settings section (see the figure below).

  4. Repeat the steps for each camera for which you want to set up these settings.

Filtering SmartFace data in Nx Witness

It is possible to filter objects based on the face attributes: Age, Gender, Face mask, Name, Id, Watchlist, IsSpoof and Score

To filter the objects

  1. Go to the right panel and select Objects.

  2. Then, type a face attribute (for example, the name of a person) based on which Nx Witness filters the results.

  3. You may further narrow your search, by specifying the time, camera, object type and selected area of the camera view.

SmartFace analytic events in Nx Witness

Nx Witness enables you to create custom rules based on SmartFace’s analytics events. After you defined an event rule based on the SmartFace events, the Nx Witness will trigger the appropriate action. For more information, see the Nx Witness documentation, the chapter Configuring Events and Actions.

In this chapter you can find information about:

SmartFace analytics event types

SmartFace Analytics Plugin provides three types of analytics events for Nx Witness VMS – PersonIdentified, PersonUnidentified and PersonIdentificationSpoofDetected.

PersonIdentified
When a person is detected, SmartFace Platform automatically performs matching against all watchlist members stored in SmartFace. When SmartFace matches (identifies) the person against a member in any of the watchlists, the system sends the analytics event PersonIdentified to Nx Witness.

PersonUnidentified
When SmartFace doesn’t match a person against any of the watchlists member (the person is unidentified), the system sends the analytics event PersonUnidentified to Nx Witness.

PersonIdentificationSpoofDetected
When SmartFace matches (identifies) the person against a member in any of the watchlists and spoof detection is configured, the system sends analytic event PersonIdentificationSpoofDetected which signals that potential face presentation attack (spoof) was detected.

Enable or disable SmartFace analytics events

  1. In the left panel, right-click the camera for which you want to enable or disable SmartFace analytics events and select Camera Settings.

  2. Go to Plugins and select SmartFace analytics plugin.

  3. To enable or disable a SmartFace analytics event, mark or clear the relevant checkboxes in the Analytic event settings section (see the figure below).

  4. Repeat the steps for each camera for which you want to enable or disable the events.

Configure rules for SmartFace analytics events

  1. Go to the Main menu (click ) and select System Administration.

  2. Click Event Rules.

  3. Click Add.

  4. In When, choose Analytics Event.

  5. In At, select the desired cameras.

  6. Choose between the event types PersonIdentified, PersonUnidentified or PersonIdentificationSpoof. Each of the events consists of a caption and description part.

  7. (Optional) You may specify the Caption contains and Description contains fields with keywords to filter out when an action is triggered for a SmartFace event. See the instructions How to fill Caption contains and Description contains below.

  8. After you defined an event rule based on the SmartFace event types, the Nx Witness will perform an action based on settings you have made in the Action panel.

How to fill Caption contains and Description contains for SmartFace analytics events

You may further specify the event by entering the keywords in the Caption contains and Description contains. How to fill in these fields depends on the type of the SmartFace event. Please check also the rules for filling out these fields. Each event consist of the Caption and the Description field.

Format of SmartFace analytics events:

PersonIdentified

  • Caption: Person < watchlist-member-display-name > matched.

    Example: Person John Doe matched.

  • Description: Person < watchlist-member-display-name > (< watchlist-member-full-name >) matched in watchlist < watchlist-display-name > (< watchlist-full-name >). Face mask status is (< face-mask-status >).MatchResultID = < match-result-id >

    Example: Person John Doe (John Doe 734939) matched in watchlist HR Employees (Human Resource Department Employees). Face mask status is (Mask present). MatchResultID = 5b7f8168-220a-43c6-9c00-87830c3c1d34

    ℹ️ <face-mask-status> can have the values No mask, Mask present or Unknown.

PersonUnidentified

  • Caption: Analyzed person (< *gender>, < age >) not matched.

    Example: Analyzed person (Female, 35) not matched.

  • Description: Person (gender: < gender >, age: < age >, face mask status: < face-mask-status >) not matched in any watchlist. *MatchResultID = < match-result-id >

    Example: Person (gender: Female, age: 35, face mask status: No mask) not matched in any watchlist. MatchResultID = 1b3ad0bc-6d81-4e72-8ed8-e1330bd34224

    ℹ️ <face-mask-status> can have the values No mask, Mask present or Unknown. < gender > can have values Male, Female or Unknown.

PersonIdentificationSpoofDetected

  • Caption: Person < watchlist-member-display-name > match spoof detected.

    Example: Person John Doe match spoof detected.

  • Description: Person < watchlist-member-display-name > (< watchlist-member-full-name >) matched in watchlist < watchlist-display-name> (<* *watchlist-full-name* *>*). Face mask status is (*<* *face-mask-status* *>*). *MatchResultID* = *<* *match-result-id* *>*

    Example: Person John Doe (John Doe 734939) matched in watchlist HR Employees (Human Resource Department Employees). Face mask status is (Mask present). MatchResultID = 5b7f8168-220a-43c6-9c00-87830c3c1d34

    ℹ️ < face-mask-status > can have the values No mask, Mask present or Unknown. < gender > can have values Male, Female or Unknown.

Rules for filling in Captions and Description contains

Please follow these rules when entering the keywords in Caption contains and Description contains:

  • Keywords are case-sensitive (Keyword is not the same as keyword)

  • Space character is used as separator (Keyword1 Keyword2)

  • Quotes can be used for multi-term keywords (“Multi term keyword”)

  • When you specify more than one keyword in either Caption or Description contains, the filter will apply to events which match with at least one of the entered keywords:

    • Example

      • The example event Description is: Person (gender: Female, age: 35, face mask status: No mask) not matched in any watchlist. MatchResultID = 1b3ad0bc-6d81-4e72-8ed8-e1330bd34224

      • Description keywords are “age: 34” “age: 35”. The filter applies to the example event as the second keyword is a part of the event Description.

      • Description keyword is 34. The filter applies to the example event as this keyword is a part of event Description (The MatchResultId near the end contains 34)

      • Description keywords are “age: 33” and “age: 34”. The filter doesn’t apply to the example event, as the keywords aren’t part of the event Description.

  • When you specify keywords for both Caption and Description contains, the filter will apply only to events, which match with both Caption and Description contains in at least one keyword:

    • Example

      • The example event Caption is: Person John Doe matched.

      • The example event Description is: Person John Doe (John Doe 734939) matched in watchlist HR Employees (Human Resource Department Employees). Face mask status is (Mask present). MatchResultID = 5b7f8168-220a-43c6-9c00-87830c3c1d34

      • Caption keyword is “John Doe” and the Description keyword is “HR Employees”. The filter applies to the example event.

      • Caption keywords are “John Doe” “Bob Doe” and the Description keywords are Management “Manual Workers”. The filter doesn’t apply to the example event, because though one Caption keyword is a part of the example event Caption, no Description keyword isn’t a part of the example event Description.

Troubleshooting for Nx Witness integration

If the integration doesn’t work properly, try the following steps:

  • SmartFace camera logs are available at %PROGRAMDATA%\Innovatrics\SmartFace where the filename corresponds to the Nx Witness camera ID.

  • There might be some issues regarding the camera rotations. To make sure that the raw frame feed is correct, check if SmartFace MPEG1 camera preview generates a valid video.

  • By default, Nx Witness Client buffers 500 ms before rendering the live preview. This means SmartFace processing must be performed under 500 ms to show bounding boxes on the live preview. If you are encounter a bigger delay than 500 ms, then you can increase the buffering to 1000 ms.
    The buffering can be set as follows:

    • In Nx Witness open the Main menu (click ).

    • Select Local Settings… and the Advanced tab.

    • To increase the buffering, move the slider Maximum Live Buffer Length to the maximum value.

    • Click OK.