Skip to main content

Data Synchronization Service (DSS)

The Data Synchronization Service (DSS) is the half of the Data Exchange Module that feeds the corridor. It reads the travelers who are due at the site from the platform's Traveler Identity Service and keeps them as watchlist members in Face Matcher, so the corridor recognizes exactly the passengers it expects. The DSS only reads from the platform and only writes to Face Matcher; it has no database and no user interface, and everything it does is set through environment variables.

What it synchronizes​

Synchronization is organized in routes. A route is one sync definition: which journeys to read, how to group them into watchlists and how often to run. On every run a route:

  1. Reads the journeys whose scheduled arrival falls in the route's time window. Invalidated journeys and journeys without a traveler are left out.
  2. Fetches the traveler of each journey and applies the route's optional identity-status condition.
  3. Keeps only travelers with a usable photo: an active biometric record with a face capture. The DSS downloads that portrait and registers it as the member's face.
  4. Creates or updates one watchlist member per traveler, links it to the route's watchlists and writes its labels.

The member ID is the traveler's reference in the Traveler Identity Service, so each traveler exists once in Face Matcher however many runs or journeys bring them in. When a traveler has several journeys in the window, the one with the latest scheduled arrival supplies the flight details. A member that has not changed since the last run is not written again.

Watchlists it manages​

By default each route keeps one watchlist per flight (perFlight), named dxm.<route>.flight.<flight number>.<yyyyMMddHHmm> from the flight number and the scheduled arrival in UTC, for example dxm.r1.flight.OS123.202611141400. The DSS creates these watchlists when it first needs them and deletes them once they are empty and expired. Alternatively a route can write every traveler into one existing watchlist (single); the DSS never creates or deletes that watchlist.

Watchlist IDs starting with dxm. are reserved for the DSS. Labels the DSS does not own, and watchlist links made by someone else, are left untouched: you can add your own labels to a synchronized member, or link it to one of your own watchlists, without the next run undoing it.

Revisions and deletion​

Each member carries the revision of its traveler and journey as labels. A run applies only newer data, so a late or repeated read never overwrites a fresher state, and a changed name, flight or portrait reaches the corridor on the next run.

Removal is a separate deletion job with its own interval. It never calls the platform: it works from what is in Face Matcher. A member leaves an expired flight watchlist once the flight's arrival is older than the configured cutoff, a member with no remaining links is deleted, and a flight watchlist is deleted once it is empty. A member that is still linked to a watchlist the DSS does not own is kept. Because deletion is independent of the platform, the corridor keeps working and keeps cleaning up during a platform outage.

Configuration​

All settings are environment variables, read once at startup. A missing or invalid value stops the service at startup, and every offending variable is named in the log. Every value is echoed in the startup log with secrets masked. To change a setting, restart the service with the new environment.

Connections and limits​

VariableDefaultMeaning
TIS_BASE_URLrequiredTraveler Identity Service URL. /traveler-identity-service is appended when the URL has no path
TIS_AUTH_MODEclientCredentialsclientCredentials (OAuth 2.0) or none for a platform with security turned off
TIS_TOKEN_URL, TIS_CLIENT_ID, TIS_CLIENT_SECRETrequired with clientCredentialsToken endpoint (<TIS>/traveler-identity-service/oauth2/token), the DSS's own client (normally corridor-dss) and its secret
TIS_PAGE_SIZE, TIS_TIMEOUT_MS, TIS_MAX_RETRIES100, 10000, 3Page size (1–100), request timeout and read retries
VPP_BASE_URL, VPP_GRAPHQL_URLrequiredFace Matcher REST API (port 8098) and GraphQL API (port 8097)
VPP_AUTH_MODE, VPP_API_KEYnoneapiKey sends the key as a bearer token
VPP_TIMEOUT_MS, VPP_MAX_RETRIES30000, 3Retries apply to reads only, never to writes
TZrequiredTime zone in which time windows are evaluated, for example Europe/Vienna
DSS_DEPLOYMENT_IDrequiredStable ID of this DSS, written to every member it owns. Never change it on restart or upgrade
DSS_MAX_CANDIDATES_PER_RUN, DSS_MAX_SNAPSHOT_ITEMSrequiredSafety limits. A run that would exceed one fails instead of working on partial data
DSS_RUN_TIMEOUT_MS900000Deadline of one run
DSS_WATCHLIST_THRESHOLD, DSS_WATCHLIST_PREVIEW_COLOR40, #4adf62Matching threshold and color of the watchlists the DSS creates
DISPLAY_LABELSfalsetrue also writes the person and flight labels the operator dashboard shows
HEALTH_LISTEN, LOG_LEVEL0.0.0.0:8660, infoListen address of the operations endpoints; log level

Routes​

Routes are numbered from 1 without gaps: ROUTE_1_*, ROUTE_2_*, and so on. An unknown ROUTE_n_* variable stops the service.

VariableDefaultMeaning
ROUTE_n_IDrequiredShort ID ([a-z0-9-], up to 16 characters); part of every watchlist name
ROUTE_n_WINDOWrequiredTime window on the scheduled arrival: TODAY, NEXT:<duration> or AROUND:-<duration>+<duration>, with durations such as 6h, 90m or 1h30m
ROUTE_n_CONDITIONSnoneOptional traveler condition: traveler.identityStatus==ENROLLED or traveler.identityStatus!=CREATED
ROUTE_n_WATCHLIST_MODEperFlightperFlight or single
ROUTE_n_DESTINATIONrequired with singleID of the existing watchlist to fill; must not start with dxm.
ROUTE_n_MAPPINGrequiredHow the member's display fields are built (see below)
ROUTE_n_SIZE_BOUNDrequiredMaximum travelers per watchlist. A run that would exceed it fails rather than truncating
ROUTE_n_SYNC_INTERVAL_MINUTESrequiredHow often the route synchronizes
ROUTE_n_DELETE_INTERVAL_MINUTESrequiredHow often the deletion job runs for the route
ROUTE_n_DELETE_OLDER_THAN_MINUTESrequiredMembers leave a flight watchlist when the arrival is older than now minus this value
ROUTE_n_WATCHLIST_THRESHOLDglobal defaultMatching threshold for this route's watchlists

ROUTE_n_MAPPING sets up to three member fields: displayName (required), fullName and note. Each is built from traveler fields (traveler.firstName, traveler.lastName, traveler.nationality, traveler.dateOfBirth, …), fields of the dominant journey (journey.conveyanceNumber, journey.routeFrom, journey.scheduledArrival, …) or quoted literals, joined with +:

ROUTE_1_MAPPING=displayName=traveler.firstName+traveler.lastName,note=journey.conveyanceNumber

Filters by nationality, terminal, gate or flight, watchlists per gate or per terminal, routes that read a traveler group instead of journeys, and event-based updates from the platform are planned. Their variables are recognized but either inactive or refused at startup.

Labels it writes​

Every synchronized member carries labels whose keys start with dxm.: who owns it (dxm.owner, dxm.deployment, dxm.route, dxm.travelerRef), the revisions it was built from, the dominant journey's flight number and times, and a trace of the last run. These labels are how the DSS recognizes its own members, and how the Corridor Event Aggregator recognizes a synchronized traveler, so never edit them. With DISPLAY_LABELS=true the DSS also writes the Watchlist Member Labels firstName, lastName, country, dob, sex, placeOfBirth, flight, route and arrival. These contain personal data and are removed again when you switch the setting off.

Operations​

All endpoints are under http://<host>:8660/corridor-data-synchronization-service. They do not authenticate the caller, so keep the port inside the corridor network.

EndpointPurpose
GET /actuator/health/liveness, GET /actuator/health/readinessProbes. A platform or Face Matcher outage does not fail them
GET /statusPer route: the last sync and deletion run with their outcome, counts and the window they used; members and watchlists that could not be resolved, with a reason; whether the platform and Face Matcher are reachable
POST /routes/{routeId}/resyncQueues an immediate sync of the route: 202 queued, 409 work already pending, 404 unknown route
GET /versionService version
GET /actuator/prometheusMetrics, such as dss_run_total, dss_members_total, dss_skipped_total, dss_unresolved_members and dss_upstream_reachable

A run ends succeeded, degraded (some members could not be written; they are listed in /status), failed or interrupted. Runs of all routes are executed one after another, and at startup each route queues a sync followed by a deletion. Run exactly one instance per corridor deployment: two would work on the same watchlists. Setup steps are in Set Up the Platform Link-up.