SmartFace Platform

How to optimize performance#

The SmartFace Platform is a flexible and scalable system. To perform it functions it uses the resources available efficiently. To keep up with the demands we are able to optimize it in several ways to provide the best service possible.

Scaling up services#

It was made in mind with scalable and distribute systems. If the amount of performed actions is higher, we can suggest scaling services.

Use GPU where helpful#

Several of the SmartFace Platform services are capable of harnessing the support of graphics cards. For more information about how to setup the graphics card and enable the GPU support please see GPU Enablement.

Configuring services to use the GPU instead of CPU#

Several of the services/containers available in the docker-compose.yml files have per default commented out several lines (the lines are starting with the # symbol). This includes lines, such as:

- Gpu__GpuEnabled=true
runtime: nvidia

To let the service use the GPU, please uncomment these lines and restart the service, either directly or by using the docker compose up -d command. It is recommended to update the name of the service and the container_name, so it is clear from the name itself whether the service is using GPU or not. Also this will allow you to set similar containers with distinct names using different configurations. Bellow is an example of the object detector service configured to use the GPU.

object-detector-GPU:
    image: ${REGISTRY}sf-object-detector:${SF_VERSION}
    container_name: SFObjectDetectorGpu
    restart: unless-stopped
    environment:
      - RabbitMQ__Hostname
      - RabbitMQ__Username
      - RabbitMQ__Password
      - RabbitMQ__Port
      - RabbitMQ__VirtualHost
      - RabbitMQ__UseSsl
      - AppSettings__Log_RollingFile_Enabled=false
      - AppSettings__USE_JAEGER_APP_SETTINGS
      - JAEGER_AGENT_HOST
      - Gpu__GpuEnabled=true
    volumes:
      - "./iengine.lic:/etc/innovatrics/iengine.lic"
    runtime: nvidia

Some of the services do support additional GPU configuration - the Tentors. The GPU card needs to support the tensors and needs to be properly configured. For more information please take a look at the GPU Enablement page. Please let the tensor to load properly after the service starts. This can cause the service to not work until the tensor mode is loaded. This might take several seconds up to few minutes depending on the configuration.

To enable the tensors, please uncomment two additional lines:

- Gpu__GpuNeuralRuntime=Tensor
- "/var/tmp/innovatrics/tensor-rt:/var/tmp/innovatrics/tensor-rt"

Below is an example of a service using GPU with the tensor as well.

pedestrian-detector-GPU:
    image: ${REGISTRY}sf-pedestrian-detector:${SF_VERSION}
    container_name: SFPedestrianDetectGpu
    restart: unless-stopped
    environment:
      - RabbitMQ__Hostname
      - RabbitMQ__Username
      - RabbitMQ__Password
      - RabbitMQ__Port
      - RabbitMQ__VirtualHost
      - RabbitMQ__UseSsl
      - AppSettings__Log_RollingFile_Enabled=false
      - AppSettings__USE_JAEGER_APP_SETTINGS
      - JAEGER_AGENT_HOST
      - Gpu__GpuEnabled=true
      - Gpu__GpuNeuralRuntime=Tensor
    volumes:
      - "./iengine.lic:/etc/innovatrics/iengine.lic"
      - "/var/tmp/innovatrics/tensor-rt:/var/tmp/innovatrics/tensor-rt"
    runtime: nvidia

Use only required video resolution#

The RTSP stream from you cameras can be processed from various resolutions. However in the default configurations higher resolutions than the Full HD (1920pxx1080px) are bringing only diminishing returns. At the same time the processing requirements are increasing drastically. Therefore it is suggested to use the Full HD as a default resolution for the RTSP Streams being processed by the SmartFace Platform.

Redetection / extraction interval#

The detection/redetection and the extraction are one of the repetetive actions that are resource demanding when being performed.

The detection (whether face, pedestrian or object detection) is being performed by the detector services. It’s purpose is to take an image (frame) and perform the detection action - find the face/pedestrian/object matching the detector’s configuration. Found faces/pedestrians/objects are then sent for further processing. For more information about detections, please take a look at the Processing features.

Extraction is a process of conversion of face features from the acquired digital image into a binary representation of the face, which is called a biometric template. For more information please take alook at Face extraction article.

Both detection and extraction intervals can be set. It is recommended to both have the same value, or the detection interval to be multiple of the extraction value, with a remainder of 0.

The default is set to 500ms for detection and 500ms for extraction. This means that both operations happen twice a second. For use cases where we need to ensure we catch each case that happens to be in front of the camera we need to make the numbers lower to have more detections per second. The default for an access control use case would be 250ms for both actions.

If we expect to have people/pedestrians/objects moving in front of the camera even faster, we need to lower the amount of miliseconds. Halving the amount each step would be generally the way to go.

To avoid too many detection actions, that are very expensive, the SmartFace Platform SmartFace Platform only performs the tracking operation in the following frames ─ keeps track of the location of the detected faces. For more information about the tracking please look at the Tracking explained.

Turn off or minimize Camera Preview quality#

The Camera preview visible in the SmartFace Station might be using considerable amount of resources, especially on a low spec machine. In such cases it is reasonable to not use the preview unless needed and to customize the preview quality. The higher quality the higher specification for the machine resources is needed. The preview quality can be set in the SmartFace Station as on the Camera configuration parameters.

• Low (height: 426 px, bit rate: 153,000)
• Medium (height: 640 px, bit rate: 450,000)
• High (height: 1280 px, bit rate: 1,400,000)

It is possible to set a custom value as well, hoever only via the REST API. For more information please read Camera preview settings.

Changing Algorithms#

The SmartFace Platform is used state of the art Neural Networks. In some cases we have more than one available for the same functionality. The different networks are specifically created to match different needs, especially to bring different answerrs to the tradeoffs between speed and accuracy. Some Neural Networks bring better results but at cost of lower efficiency and more demands on the CPU and memory of the hosting machine. That is why we offer more options for advanced users.

Changing Detection Algorithm#

The SmartFace is detecting faces from the face images. It can be done using both CPU and GPU services. We support two detection methods/algorithms: accurate, balanced. The balanced algorithm is the default. They can be changed as per camera in the REST API endpoints api/1/Cameras using the FaceDetectorResourceId parameter.

Each of them offers also it’s CPU version (value cpu), as the default is using a GPU (value gpu). The value any will let the system decide to use both CPU and GPU based on it’s own priority system.

Another option is to use the remove version of each of the parameters. Whe using the remote version, the detection is done in a standalone detector service, instead of the in camera service processing. The endpoint offers you these options:

• none
• cpu
• gpu
• accurate_cpu
• accurate_gpu
• cpu_remote
• gpu_remote
• any_remote
• accurate_cpu_remote
• accurate_gpu_remote
• accurate_any_remote

Changing Extraction Algorithm#

The SmartFace is extracting data from the face images thus generating biometric templates. Currently there is only one service that performs extraction and it is working in both CPU (SFExtractCpu service) and GPU (SFExtractGpu service) modes. Several methods/algorithms to do the extraction are supported: fast, balanced, accurate_mask, accurate_server_visa, accurate_server_wild.

The default algorithm used across the SmartFace is balanced.

The table below describes the compatibility of different algorithms across SmartFace versions.

Changing the extraction algorithm on a Windows installation#

On a Windows installation you can adjust the SmartFace.RpcExtractor.appsettings.json located in your installation directory. In a default installation it is located in C:\Program Files\Innovatrics\SmartFace. Choose another algorithm and write it instead of balanced.

"Extraction": {
    "Algorithm": "balanced"
}

After a change you need to restart the SFExtractCpu or SFExtractGpu service(s)

Changing the extraction algorithm on a Linux/Docker installation#

On a Docker/Linux installation you can easily adjust the extraction algorithm to be used in the docker-compose.yml file by adjusting the environment variable for the extraction service. To change the extraction algorithm into accurate mask, you can add this line:

- Extraction__Algorithm=accurate_mask

such as below:

extractor:
    image: ${REGISTRY}sf-extractor:${SF_VERSION}
    container_name: SFExtractCpu
    restart: unless-stopped
    environment:
      - RabbitMQ__Hostname
      - RabbitMQ__Username
      - RabbitMQ__Password
      - RabbitMQ__Port
      - AppSettings__Log_RollingFile_Enabled=false
      - AppSettings__USE_JAEGER_APP_SETTINGS
      - JAEGER_AGENT_HOST
      - Gpu__GpuEnabled=true
      - Gpu__GpuNeuralRuntime=Tensor
      - Extraction__Algorithm=accurate_mask
    volumes:
      - "./iengine.lic:/etc/innovatrics/iengine.lic"
      - "/var/tmp/innovatrics/tensor-rt:/var/tmp/innovatrics/tensor-rt"
    runtime: nvidia

To apply the changes you need to restart and/or update the extractor service. You can do so by running one of these commands:

docker-compose up -d

docker restart SFExtractCpu

Changing Object Detector Algorithm#

Similarly you can adjust the object detector algorithm. The supported values are: balanced, fast, accurate. The default value is balanced.

Changing the object detection algorithm on a Windows installation#

On a Windows installation you can adjust the SmartFace.RpcObjectDetector.appsettings.json located in your installation directory. In a default installation it is located in C:\Program Files\Innovatrics\SmartFace. Choose another algorithm and write it instead of balanced.

"Detection": {
    "Algorithm": "balanced"
}

After a change you need to restart the SFObjectDetectorCpu or SFObjectDetectorGpu service(s)

Changing the extraction algorithm on a Linux/Docker installation#

On a Docker/Linux installation you can easily adjust the object detection algorithm to be used in the docker-compose.yml file by adjusting the environment variable for the extraction service. To change the object detection algorithm into accurate, you can add this line:

Detection__Algorithm=accurate

object-detector:
    image: ${REGISTRY}sf-object-detector:${SF_VERSION}
    container_name: SFObjectDetectorCpu
    restart: unless-stopped
    environment:
      - RabbitMQ__Hostname
      - RabbitMQ__Username
      - RabbitMQ__Password
      - RabbitMQ__Port
      - RabbitMQ__VirtualHost
      - RabbitMQ__UseSsl
      - AppSettings__Log_RollingFile_Enabled=false
      - AppSettings__USE_JAEGER_APP_SETTINGS
      - JAEGER_AGENT_HOST
      - Detection__Algorithm=accurate
      # - Gpu__GpuEnabled=true
    volumes:
      - "./iengine.lic:/etc/innovatrics/iengine.lic"
    #runtime: nvidia

To apply the changes you need to restart and/or update the extractor service. You can do so by running one of these commands:

docker-compose up -d

docker restart SFObjectDetectorCpu

How to setup Authentication#

The SmartFace platform supports several methods for authentication and a security restriction.

User roles and their configuration#

The methods are using user roles and the SmartFace Station presets. For more information please read about the SmartFace Station Presets.

Setting up Keycloak#

One of the supported method is the Keycloak. It is using standard protocols and provides Single-Sign On. It is easily configurable and offers many options for advanced users.

In this guide we provide an easy basic installation allowing you to hide the SmartFace Platform behind it, exposing only public ports.

Requirements#

• 1 to N SmartFace installations, including the SmartFace Station (can be both Windows and Linux/Docker)
• Docker enabled Linux machine/virtual machine
• Machines are networked and the communication between them is allowed

Setting up the SmartFace#

• Install the SmartFace and the SmartFace Station for each machine intended for being locked behind the Keycloak

• Ensure the SmartFace Station is up and running

• Ensure the communication is enabled between the server and the Keycloak machine (allow all ports both inbound and outbound to make the installation easy. Adjust the ports after the whole setup to match only used ports)

• Open the file C:\Program Files (x86)\SmartFace Station\.env in an text editor on each SmartFace installation machine. If the SmartFace Station is running on a docker, update the .env.sfstation file.

• replace with your actual keycloak’s hostname or ip address

  ENABLE_AUTHENTICATION=true
  KEYCLOAK_ADMIN_URL=http://<your-keycloak-machine-ip-or-hostname>:8080/auth/admin
  

  • add new lines to allow mapping between the Keycloak roles and the SmartFace roles. Do not alter the lines unless you have deeper knowledge in the topic of the SmartFace access roles.

  ROLES_CLAIM_NAME=sf_roles
  ROLE_KEY_ADMIN=/admin
  ROLE_KEY_SECURITY_SUPERVISOR=/security_supervisor
  ROLE_KEY_SECURITY_OPERATOR=/security_operator
  ROLE_KEY_ACCESS_SUPERVISOR=/access_supervisor
  ROLE_KEY_ACCESS_OPERATOR=/access_operator
  ROLE_KEY_INVESTIGATION_SUPERVISOR=/investigation_supervisor
  ROLE_KEY_INVESTIGATION_OPERATOR=/investigation_operator
  ROLE_KEY_SECURITY_ADMIN=/security_admin
  ROLE_KEY_ACCESS_ADMIN=/access_admin
  ROLE_KEY_INVESTIGATION_ADMIN=/investigation_admin
  

• Restart the SmartFace Station
  • For a Windows installation: Click start, type services, open the Services application from the menu. Find the SmartFace Station, right click on the item and click restart.
  • For a Docker installation: run this command to restart the SmartFace Station docker restart SFStation, or refresh configuration via docker compose up -d

Configuring the Keycloak#

• Enable Docker and Docker Compose on a standalone Linux machine/virtual machine

• Download the keycloak-setup.zip file

• In the demo case the contents were unzipped under /srv/keycloak

• open the directory

• Ensure the file realm-export.json has read/write permissions. The rough way to do it is sudo chmod 777 realm-export.json.

• Edit the real-export.json file via bash command

  sed -i 's/localhost:3000/$Docker_hostname/g' realm-export.json
  

  where the $Docker_hostname matches the hostname of the machine where the Keycloak docker runs. For example for our demo it is: sed -i 's/localhost:3000/sface-kclk-3u:3000/g' realm-export.json

  $Docker_host_ip - IP address of the host machine that runs the docker-compose command $AS_hostname - Hostname where SmartFace Station can be found $Docker_hostname - Hostname under which the gatekeeper will be accessible on the docker host machine sface-docker-1u:3000

• Edit the docker-compose.yml file

  • We name our Keycloak service/container in the docker-compose.yml. In our case such as keycloak-server.10.11.64.69.nip.io:

    keycloak-server.10.11.64.69.nip.io:
    build: .
    ports:
      - 8080:8080
    

  • For each SmartFace installation we add a new gatekeeper container:

    • In our demo there is code for 2 containers, one per each gatekeeper, connecting to 2 machines sface-kclk-1w and sface-kclk-2w

    • sface-kclk-2w

      • Will use port 3002. Each gatekeeper should have it’s own port. This is the port used for the outside world.

        ports:
              - 3002:3000
        

      • SmartFace Station is located at (from the point of view of the Keycloak machine)

        - --upstream-url
        - http://10.11.64.70:8000
        

      • We set the client-id as follows ( i apologize for confusing name, it should rather be a general name without having a specific’s server name in there). This is something that can be setup in the Keycloak admin web site along with other things such as the secret and the group roles.

        - --client-id
        - smartgate-on-sface-demo-1w
        

      • The URL of the Keycloak server. In our case we use the URL of the same machine that runs the gatekeeper

      - --discovery-url
      - http://keycloak-server.10.11.64.69.nip.io:8080/auth/realms/master
      

      • Setup the client secret matching the client defined in the client id. I recommend to use the default client and it’s client id (per default called smartgate-on-sface-demo)

      - --client-secret
      - '123abc12-345-defg-123a-123abc123abc'
      

      • We set the dependency on the container with the Keycloak. In our demo case the container for the keycloak server is names as below:

      depends_on:
            - 'keycloak-server.10.11.64.69.nip.io'
      

    • Full sample code below for 2 gatekeepers:

      # Keycloak-gatekeeper
      # - reverse-proxy that will sit in-front of our frontend server
      keycloak-gatekeeper-sface-kclk-2w:
        # See https://github.com/keycloak/keycloak-gatekeeper#getting-started
        image: quay.io/keycloak/keycloak-gatekeeper:9.0.3
        ports:
          - 3002:3000
        command:
          # gatekeeper will listen on port 3000, so you need to open http://localhost:3000/ in your browser
          - --listen
          - :3000
          # gatekeeper will proxy server (SmartFace Station FES) running on host on port 8000. This solution works in
          # docker-for-macosx. We need to find solution for linux also.
          - --upstream-url
            # sface-kclk-2w
          - http://10.11.64.70:8000
          # This client-Id is set in our `realm-export.json`
          - --client-id
          - smartgate-on-sface-demo-1w
          # Keycloak server, that will be used by this gatekeeper
          - --discovery-url
          - http://keycloak-server.10.11.64.69.nip.io:8080/auth/realms/master
          # This client-secret is set in our `realm-export.json`
          - --client-secret
          - '123abc12-345-defg-123a-123abc123abc'
          # This is needed when `enable-refresh-tokens=true`. So not needed for us in develop. But I'll keep it here.
          # Must be a valid AES-128/AES-256 key of either 16 or 32 characters.
          - --encryption-key
          - vGcLt8ZUdPX5fXht
          # For development we dont use SSL, so we MUST disable 'Secure' flag in Cookies
          - --secure-cookie=false
          # We want to see more debug info in console
          - --enable-logging
          # First, we disabled 'refresh-tokens', because gatekeeper created a very large refresh-token Cookies,
          # and then it just return `HTTP 431 Request Header Fields Too Large`.
          # ...
          # But we have found, that by removing unnecessary 'defaultClientScopes' from our Keycloak client,
          # the token size (and therefore Cookie size) is smaller.
          # For our purposes, we need these client-scopes: "sface-demo-1w-scope", "profile", "email"
          - --enable-refresh-tokens=true
        depends_on:
          - 'keycloak-server.10.11.64.69.nip.io'
        # There is no easy way in docker-compose to wait until keycloak-server starts.
        #
        # gatekeeper starts very quickly. So it makes some requests, and stops after unsuccessful
        # connection. So we need to auto-restart and wait, until keycloak-server is ready.
        restart: always
      
      keycloak-gatekeeper-sface-kclk-1w:
        # See https://github.com/keycloak/keycloak-gatekeeper#getting-started
        image: quay.io/keycloak/keycloak-gatekeeper:9.0.3
        ports:
          - 3001:3000
        command:
        # gatekeeper will listen on port 3000, so you need to open http://localhost:3000/ in your browser
          - --listen
          - :3000
        # gatekeeper will proxy server (SmartFace Station FES) running on host on port 8000. This solution works in
        # docker-for-macosx. We need to find solution for linux also.
          - --upstream-url
        # - http://host.docker.internal:8000
        # sface-kclk-2w
          - http://10.11.64.71:8000
        # This client-Id is set in our `realm-export.json`
          - --client-id
          - smartgate-on-sface-demo-1w
        # Keycloak server, that will be used by this gatekeeper
          - --discovery-url
          - http://keycloak-server.10.11.64.69.nip.io:8080/auth/realms/master
        # This client-secret is set in our `realm-export.json`
          - --client-secret
          - '3d9e08d7-7d7c-442d-880b-fbbf9fae4263'
        # This is needed when `enable-refresh-tokens=true`. So not needed for us in develop. But I'll keep it here.
        # Must be a valid AES-128/AES-256 key of either 16 or 32 characters.
          - --encryption-key
          - vGcLt8ZUdPX5fXht
        # For development we dont use SSL, so we MUST disable 'Secure' flag in Cookies
          - --secure-cookie=false
        # We want to see more debug info in console
          - --enable-logging
          - --enable-refresh-tokens=true
        depends_on:
          - 'keycloak-server.10.11.64.69.nip.io'
        restart: always
      

  Run the docker containers using the command

  docker compose up -d
  

Running the GateKeeper#

• Visit the Keycloak’s server via URL:

  http://<your-keycloak-ip-or-hostname>:8080/auth/
  

• Click on Administration Console

• Use admin/admin credentials in the default setup to login

• Manage groups

  • Click in the left menu: Manage>Groups
  • Click New and add each group matching the setup in the .env files.

  /admin
  /security_supervisor
  /security_operator
  /access_supervisor
  /access_operator
  /investigation_supervisor
  /investigation_operator
  /security_admin
  /access_admin
  investigation_admin
  

• Manage clients

  • In our demo we manage the smartgate-on-sface-demo-1w client. Per default called smartgate-on-sface-demo.
  • Open Clients. In the Settings tab adjust the URLs and URIs as needed. Add as many URIs with correct ports matching the docker-compose.yml gatekeepers as needed.

  

  • In the Credentials tab, the secret should match the secret used in the docker-compose.yml and realm-export.json
  • Client scopes should be matching

  

  • Mappers tab, there should be a mapper created. The Token Claim Name needs to match the setup from the .env file, in our case it’s sf_roles

  

Securing the communication#

Through the above configuration we have basically created secured channels to communicate with the SmartFace Station. To ensure that only the selected ports are used we should disable any ports that are normally used to communicate with the SmartFace Station and the SmartFace Station. This can be done in the firewall configuration and you should disable the unused ports between the SmartFace installation machines and the outer world, especially the ports 8000, 8098, 8097. Disable unused ports between the SmartFace installation machines and the Keycloak machine.

We also recommend to change default admin user in the Keycloak.

Using the configured Keycloak#

Via browser navigate to your keycloak server under defined port of the gatekeeper. In our case when entering the port 3001 we access the SmartFace machine #1, when entering the port 3002 we access the SmartFace machine #2.

How to configure smart camera#

SmartFace Platform offers you the capability to configure SFE Stream Processor via REST API. This chapter provides you with information about configuring the SFE Stream Processor using the provided API endpoint by SmartFace Platform.

The configuration of the SFE Stream Processor can be accessed via an API endpoint /api/v1/EdgeStreams

It’s essential to understand the parameters required for configuration. Refer to the SFE Stream Processor configuration documentation for a comprehensive description of parameters.

Below is an example JSON file and a table outlining parameters and their possible values:

{
  "face_detection": {
    "enable": true,
    "max_detections": 10,
    "min_face_size": 20
    "max_face_size": 200
    "order_by": "DetectionConfidence",
    "tracking": {
      "output_threshold": 1500,
      "input_threshold": 1000,
      "stability": 85,
      "max_frames_lost": 25
    },
    "crop": {
      "enable": true,
      "max_size": 50.0,
      "image_format": "Raw",
      "image_quality": 0.85
    }
  },
  "face_extraction": {
    "enable": true,
  },
  "face_liveness_passive": {
    "enable": true,
    "strategy": "OnEachExtractedFace",
    "conditions": [
      {
        "parameter": "FaceSize",
        "lower_threshold": 30.0
      },
      {
        "parameter": "FaceRelativeArea",
        "lower_threshold": 0.009
      },
      {
        "parameter": "FaceRelativeAreaInImage",
        "lower_threshold": 0.9
      },
      {
        "parameter": "YawAngle",
        "lower_threshold": -20.0,
        "upper_threshold": 20.0
      },
      {
        "parameter": "PitchAngle",
        "lower_threshold": -20.0,
        "upper_threshold": 20.0
      },
      {
        "parameter": "Sharpness",
        "lower_threshold": 0.6
      }
    ]
  },
  "face_identification": {
    "enable": true,
    "threshold": 40
  }
  "messaging": {
    "enable": true,
    "strategy": "OnNewAndInterval",
    "interval": 250,
    "allow_empty_messages": false
  },
  "full_frame": {
    "image_width": null,
    "image_height": null,
    "image_format": "Raw",
    "image_quality": 0.85
  },
  "log": {
    "level": "Debug",
    "path": "dev.log"
  },
  "license": {
    "data": ""
  }
}