Skip to main content

Leader and Follower Setup

Leader and Follower watchlist synchronization lets one central Face Matcher instance (the Leader, at headquarters or in the cloud) manage the watchlists for any number of site instances (the Followers). Every Follower keeps a full local copy of the watchlists and matches locally, so a site keeps identifying people when the link to the Leader is down and catches up when it returns. Watchlists are edited on the Leader only. The concept is introduced in Watchlists.

Components​

ComponentWhereRole
Watchlist update-log streamLeader site, RabbitMQ streams plugin (port 5552, RabbitMQ__StreamsPort)Journal of every watchlist change (member registered, updated or removed, watchlist assignments) in chronological order
db-synchronization-leaderLeader siteServes the journal to Followers over gRPC; host port 8100, HTTP/2 (Kestrel__EndpointDefaults__Protocols=Http2)
db-synchronization-followerEach Follower sitePolls the Leader at Leader__Address, applies the changes to the local database in the same order and copies member images into the local S3 storage

Leader and Follower synchronization: the Leader service reads the watchlist update log from RabbitMQ and answers gRPC requests from the Follower service, which applies the changes to its local database

Run the same Face Matcher version and the same extraction algorithm on all sites: face templates are copied as-is and are bound to the model version.

Install the Leader​

  1. Start from a standard installation. The Leader service reads the database, RabbitMQ and S3 settings from .env like every other service, so nothing extra is needed there.
  2. Optional but recommended: enable authentication. db-synchronization-leader shares the Authentication__* settings of api and graphql-api (section 3.3 of .env), so switching on OAuth2 for the APIs also protects the Leader endpoint with the same authority; see Authentication.
  3. Expose the gRPC endpoint. The release Compose file publishes it as 8100:${Hosting__Port}. Open that port to the Follower sites only, and terminate TLS with a valid certificate in front of it (see HTTPS); the Followers then use an https:// address.
  4. Start the service and stop the Follower container, which the package starts by default: docker compose up -d db-synchronization-leader && docker compose stop db-synchronization-follower.
  5. If the release notes ask for it, or Followers do not receive existing members, rebuild the journal from the database with ./populate-wl-update-log-stream.sh. The database stays the source of truth; the script only regenerates the stream.

Install a Follower​

A Follower holds only the watchlist data it receives from the Leader. All watchlists and watchlist members that exist on the instance before synchronization starts are deleted. Start from an empty instance, or export what you need first.

  1. Set section 3.21 of .env. Leader__Address is the Leader's gRPC endpoint as reachable from this site. The ClientAuthentication__* values are needed only when the Leader requires authentication (client-credentials flow; Audience is required with Auth0 and may stay empty otherwise):

    Leader__Address=https://leader.example.com:8100
    Leader__RequestBatchSize=1000
    ClientAuthentication__UseAuthentication=true
    ClientAuthentication__TokenEndpoint=https://auth.example.com/oauth/token
    ClientAuthentication__ClientId=<client id>
    ClientAuthentication__ClientSecret=<client secret>
    ClientAuthentication__Audience=
  2. Make the watchlists read-only on the REST API with FeatureManagement__ReadOnlyWatchlists=true in .env (section 3.2). This disables PUT, POST and DELETE on the watchlist and watchlist member endpoints, so local edits, including those from Station, are rejected.

  3. Keep watchlist autolearn off. Detections are not sent back to the Leader, so faces learned locally would be lost; autolearn is off by default and is switched in Station settings or through the REST API PUT /api/v1/Setup/Watchlists/AutoLearn.

  4. Start the Follower and stop the Leader container on this site: docker compose up -d db-synchronization-follower && docker compose stop db-synchronization-leader.

  5. Watch the synchronization with docker compose logs -f db-synchronization-follower, then check in Station that the Leader's watchlists appear.

Limitations​

  • Watchlists are managed on the Leader only; Followers serve them read-only.
  • Only watchlists are synchronized. Detections, events and face search data stay on the site that produced them.
  • A Follower is single-tenant. The Leader may be single- or multi-tenant.
  • Matching is always local, so a Follower keeps working during a Leader outage and applies the missed changes afterwards.
  • The same update-log stream also feeds edge devices; see Edge watchlist synchronization.