Watchlists

Introduction#

Watchlist is a SmartFace entity that represents a database or a list of a persons, called a Watchlist Members. One Watchlist Member can be added to one or many Watchlists.

Watchlist can be a nation-wide register of citizens, list of employees, attendees of an event or venue, students of a given university or class members.

You can have one or multiple Watchlists in SmartFace. You can put all persons in a single Watchlist or organize them into partitioned structure based on your needs.

For example you can put all employees within a single Watchlist or you can organize them into watchlist per departnemt. Similair to school attendance use case you can put all students into single watchlist or alternativelly you can organize them into watchlist per class schema.

Inputs#

For Enrollment or Search the SmartFace accepts an image that meets certain requirements. Currently, the following file formats are supported:

• Windows bitmaps - *.bmp, *.dib
• JPEG files - *.jpeg, *.jpg, *.jpe
• JPEG 2000 files - *.jp2
• Portable Network Graphics - *.png
• WebP - *.webp
• Portable image format - *.pbm, *.pgm, *.ppm *.pxm, *.pnm
• TIFF files - *.tiff, *.tif

Enrollment#

Process of adding a person into a watchlist is called an Enrollment. Person can be enrolled via the SmartFace Station or programmatically via SmartFace REST API.

Enrollment via API#

Enrollment to Watchlist is avaialble in REST API on the   POST   /api/v1/Watchlists/Register endpoint and this is the sample of minimum request to enroll new member:

{
  "id": "someUniqueId",
  "images": [{
    "data": "/9j/4AAQ..........................f/9k="
  }],
  "watchlistIds": ["060f1083-6610-4116-9082-d8e26596a115"],
  "fullName": "John Smith"
}

Full request is documented in the REST API section.

Search#

SmartFace provides a feature to search by the picture in your Watchlists. You provide an image with a face (one or many), SmartFace compares faces with all Watchlist Members and returns list of Watchlist Members that fits search criteria best. This type of search is called an Identification in biometry. Watchlist Search is currently available only programatically via SmartFace API.

Search via API#

The Watchlist Search is available in REST API on the   POST   /api/v1/Watchlists/Search endpoint and this is the sample of a minimum request to search all Watchlists with matching threshold of 50 and with one best matching candidate as result:

{
  "image": {
    "data": "/9j/4AAQ..........................f/9k="
  },
  "threshold": 50,
  "maxResultCount": 1
}

Full request is documented in the REST API section.

Multi-site Watchlists Synchronization#

The SmartFace Platform version 4.25 introduces an advanced functionality Multi-site Watchlists Synchronization which is a part of Leader - Follower architecture. Read more in Deployment guide.

Watchlist Synchronization To Edge Streams#

The SmartFace Platform version 4.22 has introduced an advanced feature that enables the synchronization of Watchlist Members with Edge streams. Edge streams are typically deployed on embedded devices such as smart cameras, and they use a synchronized database to identify people directly on the hardware. The synchronization process involves sending identification results to Edge streams using a specific data structure called frame_data.

Required version of the SmartFace Embedded Stream Processor is 2.5 or higher.

Data Synchronization#

• Edge streams databases do not contain information about watchlists, unlike the SmartFace.
• SmartFace has many (M:N) relationship between watchlists and Watchlist Members, but only Watchlist Members are synchronized.
• SmartFace selects and synchronizes all Watchlist Members associated with specific watchlists
• Synchronization is achieved using a proprietary protocol based on proto messages via MQTT.
• The protocol is designed to handle unexpected interruptions during synchronization and efficiently restore the Edge streams database even after extended periods of inactivity.

The following data is synchronized to Edge streams:

• Array of templates belonging to Watchlist Members.
• WatchlistMember internal ID represented as 32 hexadecimal digits separated by hyphens: 00000000-0000-0000-0000-000000000000
• Data are provided in key/value pairs, including member display names, external IDs, and labels.

Data source#

Synchronization is based on the journal of Watchlist Member delta updates stored in a RabbitMQ stream, which we’ll refer to as WLStream for the rest of this document. WLStream serves as a durable queue for multiple clients, enabling seeking without destructive consumption.

Components#

There are two key components involved in watchlist synchronization:

• EdgeStreamStateSynchronizer (ESSS): This component communicates with Edge streams via MQTT through a proprietary protocol. It synchronizes delta updates from the WLStream journal to Edge streams.

• EdgeStreamProcessor: This component communicates with Edge streams via MQTT indirectly (messages are forwarded from MQTT to a standard RabbitMQ queue). It receives FrameData messages with pre-calculated data from Edge streams, including face templates, identification results, and liveness information for detected faces.

Watchlist synchronization process#

1. Registration: To add a member to the watchlist, use the REST API to register the member.

2. Update In-Memory Database: The member’s data is then sent to the WATCHLIST_UPDATER exchange. This exchange updates the in-memory database of the WatchlistMatcher component.

3. Stream Data: Simultaneously, the member data is streamed through WLStream, which is a RabbitMQ stream. It’s written to WLStream as a single delta update.

4. Synchronize to EdgeStream: ESSS (EdgeStream Synchronization Service) synchronizes the member data to Edge streams.

5. Face Detection and Identification: Within Edge streams, face detection is performed, and a face template is calculated. Identification is then carried out using this data. The calculated information is sent to MQTT as a FrameData message.

6. Process FrameData: EdgeStreamProcessor reads the FrameData message, and based on the configuration settings, it proceeds with standard processing. This may include generating notifications and saving relevant data to the database.

Installation#

Watchlist synchronization is disabled by default, as it primarily benefits customers using Edge streams. To enable this feature, follow these steps:

1. Ensure that the ESSS component is included in your Docker Compose or orchestration mechanism.

2. Create a WLStream instance and populate it with Watchlist Members from your SQL database using the sfadmin populate-wl-update-log-stream command.

Example command to populate WL stream in linux environment using docker:

docker run --network sf-network -it --rm registry.gitlab.com/innovatrics/smartface/sf-admin:v5_4.22.0 populate-wl-update-log-stream --rmq-user xyz --rmq-pass xyz --rmq-host rmq --rmq-virtual-host / --rmq-port 5672 --rmq-streams-port 5552 --db-engine PgSql --connection-string "Server=pgsql;Database=smartface;Username=postgres;Password=Test1234;Trust Server Certificate=true;" --rmq-use-ssl false

Example command to populate WL stream on windows environment:

.\sfadmin.exe populate-wl-update-log-stream --rmq-user xyz --rmq-pass xyz --rmq-host localhost --rmq-virtual-host / --rmq-port 5672 --rmq-streams-port 5552 --db-engine MsSql --connection-string "Server=localhost\sfexpress;Database=smartface;UID=smartface;Password=smartface;Timeout=60;TrustServerCertificate=true;Application Name=EntityFramework" --rmq-use-ssl false

For user name and password, use your actual login data

Configuration#

ESSS Configuration#

Modify the following environment variables in your ESSS configuration:

• MQTT broker configuration (hostname, username, password, port, SSL usage).
• RabbitMQ streams plugin port.
• Batch settings for reading updates from the log stream.
• MQTT RPC timeout.

# EdgeStreamsStateSynchronizer

# MQTT broker config (by default this is RMQ broker with MQTT plugin enabled)
MQTT__Hostname=rmq
MQTT__Username=guest
MQTT__Password=guest
MQTT__Port=1883
MQTT__UseSsl=false

# RMQ streams plugin port
RabbitMQ__StreamsPort=5552

# Number of messages from update log stream to read at once
SynchronizationConfig__UpdateLogReadBatchCount=1000
# Maximum time in ms to wait if there is not enough messages to fill whole batch size, after this time batch will be trigger anyways
SynchronizationConfig__BatchTriggerMs=1000

# MQTT RPC timeout
Rpc__TimeoutMs=10000

EdgeStreamProcessor Configuration#

Configure the behavior of EdgeStreamProcessor with the following settings:

MatchingDataStrategy determines how the SmartFace Platform handles identifications conducted in the EdgeStream environment:

• EdgeStreamOnly Mode: In this mode, EdgeStreamProcessor exclusively relies on Edge stream for identifications and doesn’t attempt identification on the server hardware. Instead, it makes RPC (Remote Procedure Call) requests to retrieve business data related to the identified member, such as DisplayName and relevant Watchlist information, to enable notifications and data storage with the highest quality. There are specific scenarios to consider:

  • If the RPC request to acquire business data fails or the data is unavailable, identifications from Edge stream are not used for further processing.

  • If the matching score falls below the obtained Watchlist Threshold, identifications from Edge stream are not considered for further processing.

  • Notifications in the SF Platform are generated based on data from the SF Platform database, rather than relying on identification metadata received from Edge stream.

  • Edge stream might return multiple identifications according to its configuration; in such cases, SF Platform selects the candidate with the highest score for further processing.

• ServerOnly Mode: In ServerOnly mode, EdgeStreamProcessor disregards the identification information provided in the FrameData message. Instead, it independently conducts identifications on the server hardware using the WatchlistMatcher component. This mode ensures that identifications are solely performed within the server environment.

LivenessDataStrategy determines how the SmartFace Platform handles liveness detection within the EdgeStream environment:

• EdgeStreamOnly Mode: In this mode, EdgeStreamProcessor exclusively relies on Edge stream for liveness detection and doesn’t attempt liveness checks on the server hardware. EdgeStreamProcessor searches for pre-calculated liveness data only for the liveness types configured via the API. If no liveness data is received from Edge stream for the configured liveness type (e.g., distant or nearby), the spoof check is considered “NotPerformed.” Other important points to note include:

  • ConditionStrings defined in SF via the API are ignored. EdgeStreamProcessor respects the evaluation of conditionString that is defined and performed within Edge stream.

  • Processing units of resource IDs (such as CPU, GPU, or any) are disregarded. Only the evaluation of liveness type takes precedence.

• ServerOnly Mode: In ServerOnly mode, EdgeStreamProcessor disregards the liveness information provided in the FrameData message and independently conducts liveness checks on the server hardware. It’s important to keep in mind that to support this server mode, Edge stream should send a face crop with a FaceArea size greater than or equal to 5. This ensures that liveness checks are performed exclusively within the server environment.

Templates - In the SmartFace Platform, Edge stream has the capability to pre-calculate face templates using edge hardware. if face templates are not already included in the data received from Edge stream, SF Platform will ensure their extraction on the server-side. This ensures that face templates are consistently available for further processing, regardless of the specific configuration or strategy chosen within the SmartFace Platform.

Enabling watchlist synchronization#

Once your environment is set up and configured, you can enable synchronization for each Edge stream individually via the SmartFace Station or API. Use the PUT /api/v1/EdgeStreams/{id}/WatchlistSynchronization endpoint with the desired watchlistSyncOption value:

• None - synchronization is turned off, and existing synchronized data remains.
• All - synchronize members from all watchlists.
• Selected - synchronize members only from watchlists listed in watchlistIds array (since version v5_4.23)

Array watchlistIds was added id version v5_4.23. It may contain only ids of existing watchlists or it could be empty.

Maintenance#

As the journal log can grow in size over time, it’s important to perform regular maintenance. To compact the log data, use the sfadmin populate-wl-update-log-stream command. This command creates a new WLStream with an incremented generation ID and deletes the old one.

Make sure to run this command when only dependency containers (SQL, MinIO, RMQ) are running, and SmartFace components are turned off.