Installation

In this section, you will find instructions on how to install the SFE Stream Processor on various platforms:

You can use the SFE Stream Processor Client applications to test the SFE Stream Processor is configured and running correctly.

We also provide SmartFace Embedded Camera plugin for Nx Witness Integration VMS


Lilin AI camera installation

Download the latest version of the SFE Stream Processor Lilin camera plugin at our Customer Portal.

Install SmartFace Embedded on your camera

After you unpack the package you can upload and install the plugin (plugincv22s66.bin) on your camera with Ambarella CV22 chip and Ambarella SDK v3.0.

  1. Access the camera at the camera’s IP address/URL and log in.

  1. Go to Settings->Maintenance->Firmware update and click Choose File to choose the plugincv22s66.bin file from your file system. Click Submit to upload and install the plugin.

  1. When the plugin is installed you will be redirected to the Settings->LPKG menu where you will see Innovatrics SmartFace Embedded Plug-In. To run the plugin click the checkbox in the top left corner. When the plugin is running the Status changes to Running.

  1. To stop the plugin uncheck the checkbox. When the plugin is not running the Status changes to Stopped.

  2. SmartFace Embedded configuration can be found at <camera_IP_address> port 8592.

Get the HardwareID and install the license

To generate and deploy the license follow the Licensing instructions

Configure SFE Stream Processor

Refer to the SFE Stream Processor documentation for more information.

SmartFace integration

To receive notifications from SmartFace Embedded Stream Processor in SmartFace Station you have to configure the SFE Stream Processor connection and create an Edge stream in SmartFace Station Follow the instruction to configure SmartFace Platform.

Supported camera models

The SFE Stream Processor Lilin camera plugin supports all Lilin AI series cameras with Ambarella CV22 chip and Ambarella 3.0.x firmware, for example:


Hanwha camera installation

Download the latest version of the SFE Stream Processor Hanwha camera plugin at our Customer Portal.

Install SmartFace Embedded on your camera

After you unpack the package you can upload and install the Hanwha Open Platform Application (sfe_stream_processor.cap) on your camera with Ambarella CV22 chip and Ambarella SDK v3.0.

  1. Access the camera at the camera’s IP address/URL and log in.

  1. Go to Settings->Open platform->Open platform and click to choose sfe_stream_processor.cap file from your file system. Click Install to upload and install the application.

  1. When the application is installed you will see the new application sfe_stream_processor installed. To run the application click Start button. When the application is running the Status changes to Running....

  1. When the application is running you can also check the CPU and memory utilization as well as the run time of the application by clicking on the Health button.

  2. To stop the application uncheck the checkbox. When the application is not running the Status changes to Stopped.

  3. SmartFace Embedded configuration can be found at <camera_IP_address>:8592. You can also click on Go App button to open the configuration.

  4. To uninstall the application click on Stop button and then on Uninstall button.

Get the HardwareID and install the license

To generate and deploy the license follow the Licensing instructions

Configure SFE Stream Processor

Refer to the SFE Stream Processor documentation for more information.

SmartFace integration

To receive notifications from the SmartFace Embedded Stream Processor in SmartFace Station you have to configure the SFE Stream Processor connection and create an Edge stream in SmartFace Station Follow the instructions to configure SmartFace Platform.

Supported camera models

The SFE Stream Processor Hanwha camera application currently supports all Hanwha P series AI cameras with Ambarella CV2 chip, for example:


Axiomtek (Hailo) installation

Installation instructions for SFE Stream Processor on Axiomtek RSC101 with Hailo-8 chip.

Download the latest version of the SFE Stream Processor Axiomtek package at our Customer Portal.

Login to RSC101

Use default username test and password 123456

Upgrade Ubuntu

First, you need to update Ubuntu to version 22.04, which is required by the latest version of Hailo drivers

Install HailoRT drivers

Install the following packages from the Hailo developer zone:

  • HailoRT – PCIe driver Ubuntu package (deb) V4.12.1
  • HailoRT – Ubuntu package (deb) for amd64 V4.12.1 For more information see this page

Before installing the drivers you need to install the correct version of Linux headers or Intel IoTG:

$ sudo apt install linux-headers-5.15.0-1026-intel-iotg

Install PCIe driver

$ sudo dpkg --install hailort-pcie-driver_4.12.1_all.deb

Install HailoRT libraries

$ sudo dpkg --install hailort_4.12.1_amd64.deb

PC restart is required after driver installation.

After boot, you can use the hailortcli tool to validate that the device is identified:

$ hailortcli scan

Install H.264 decoder for GStreamer

Install the gstreamer1.0-plugins-bad package

$ sudo apt install gstreamer1.0-plugins-bad

Install VA-API drivers

Install VA-API to enable hardware video acceleration capabilities

$ sudo apt-get -y update
$ sudo apt-get -y install va-driver-all vainfo

Verify installation:

$ sudo vainfo

Note: The command should output a list of supported profiles and entry points

(Optional) Install the Intel GPU utilization tool

$ sudo apt-get -y install intel-gpu-tools

and display intel GPU utilization (needs root privileges):

$ sudo intel_gpu_top

Install VA-API GStreamer support

$ sudo apt-get -y install gstreamer1.0-vaapi

Verify that you have access to VA-API elements

$ gst-inspect-1.0 | grep vaapi

you should see something like this:

vaapi: vaapidecodebin: VA-API Decode Bin
vaapi: vaapih264dec: VA-API H264 decoder
vaapi: vaapih264enc: VA-API H264 encoder
# ... more elements

Run your first GStreamer pipeline with VA-API acceleration

$ gst-launch-1.0 filesrc location=test.mp4 \
! qtdemux ! vaapidecodebin ! vaapipostproc ! xvimagesink

you should see that in intel_gpu_top, the Video line shows some activity

Get the HardwareID and install the license

To generated and deploy the license follow the Licensing instructions.

Verify the installation by running SFE Stream Processor and local MQTT broker

Run MQTT broker

$ bin/mqtt_broker

Use run.sh to run the SFE Stream Processor with the sample video file

$ ./run.sh

Use sfe_client_gui or sfe_client_cli to consume and visualize MQTT messages and register new users in DB.

Run bin/sfe_client_gui. See sfe_client_gui_example.png for configuration.

Run sfe_client_cli to consume MQTT messages:

$ ./bin/sfe_client_cli setting/settings_usb.yaml listen --crop

to save crop image and template of the latest detected face:

$ ./bin/sfe_client_cli setting/settings_usb.yaml listen --crop --save-crop crop.jpg --save-template template.tpl

to register a new user with a template:

$ ./bin/sfe_client_cli setting/settings_usb.yaml register-user Lipo template.tpl

Using multiple settings files to process multiple streams

With the sfe_stream_processor application, it is easy to run multiple streams by just duplicating settings.yaml file and changing parameters of the frame_input solver in settings.yaml. Then, you can run the application similarly to this:

./bin/sfe_stream_processor settings.yaml settings2.yaml ... settingsN.yaml 

A specific example:

settings_usb.yaml:

...
solvers:
  frame_input:
    solver: ./solver/gstreamer_input.cpu.solver
    parameters:
      - name: gst_pipeline
        value: "v4l2src ..."
      - name: gst_width
        value: 1280
      - name: gst_height
        value: 720
...

settings_rtsp.yaml:

...
solvers:
  frame_input:
    solver: ./solver/gstreamer_input.cpu.solver
    parameters:
      - name: gst_pipeline
        value: "rtspsrc ..."
      - name: gst_width
        value: 1280
      - name: gst_height
        value: 720
...

settings_video.yaml:

...
solvers:
  frame_input:
    solver: ./solver/gstreamer_input.cpu.solver
    parameters:
      - name: gst_pipeline
        value: "filesrc ..."
      - name: gst_width
        value: 1280
      - name: gst_height
        value: 720
...

running:

./bin/sfe_stream_processor setting/settings_usb.yaml setting/settings_rtsp.yaml setting/settings_video.yaml

SmartFace integration

Update the SmartFace Platform to the latest version available at Github. The Linux Docker installation steps are documented here.

To receive notifications from SmartFace Embedded Stream Processor in SmartFace Station you have to configure the SFE Stream Processor connection and create an Edge stream in SmartFace Station Follow the instructions to configure SmartFace Platform.

Supported devices

The SFE Stream Processor supports all edge devices with Hailo-8 AI accelerator, for example:


NVidia Jetson installation

Installation instructions for SFE Stream Processor on Jetson devices (Xavier NX, AGX Xavier, Orin)

Download the latest version of the SFE Stream Processor Jetson package at our Customer Portal.

Get the HardwareID and install the license

To generated and deploy the license follow the Licensing instructions.

Verify the installation

Verify the installation by running SFE Stream Processor and local MQTT broker

Run MQTT broker

$ bin/mqtt_broker

Use run.sh to run the SFE Stream Processor with the sample video file

$ ./run.sh

Use sfe_client_gui or sfe_client_cli to consume and visualize MQTT messages and register new users in DB.

Run bin/sfe_client_gui. See sfe_client_gui_example.png for configuration.

Run sfe_client_cli to consume MQTT messages:

$ ./bin/sfe_client_cli setting/settings_usb.yaml listen --crop

to save crop image and template of the latest detected face:

$ ./bin/sfe_client_cli setting/settings_usb.yaml listen --crop --save-crop crop.jpg --save-template template.tpl

to register a new user with a template:

$ ./bin/sfe_client_cli setting/settings_usb.yaml register-user Lipo template.tpl

Using multiple settings files to process multiple streams

With the sfe_stream_processor application, it is easy to run multiple streams by just duplicating settings.yaml file and changing parameters of the frame_input solver in settings.yaml. Then, you can run the application similarly to this:

./bin/sfe_stream_processor settings.yaml settings2.yaml ... settingsN.yaml 
A specific example:

settings_usb.yaml:

...
solvers:
  frame_input:
    solver: ./solver/gstreamer_input.cpu.solver
    parameters:
      - name: gst_pipeline
        value: "v4l2src ..."
      - name: gst_width
        value: 1280
      - name: gst_height
        value: 720
...

settings_rtsp.yaml:

...
solvers:
  frame_input:
    solver: ./solver/gstreamer_input.cpu.solver
    parameters:
      - name: gst_pipeline
        value: "rtspsrc ..."
      - name: gst_width
        value: 1280
      - name: gst_height
        value: 720
...

settings_video.yaml:

...
solvers:
  frame_input:
    solver: ./solver/gstreamer_input.cpu.solver
    parameters:
      - name: gst_pipeline
        value: "filesrc ..."
      - name: gst_width
        value: 1280
      - name: gst_height
        value: 720
...

running:

./bin/sfe_stream_processor setting/settings_usb.yaml setting/settings_rtsp.yaml setting/settings_video.yaml

SmartFace integration

Update the SmartFace Platform to the latest version available at Github. The Linux Docker installation steps are documented here.

To receive notifications from SmartFace Embedded Stream Processor in SmartFace Station you have to configure the SFE Stream Processor connection and create an Edge stream in SmartFace Station Follow the instructions to configure SmartFace Platform.

Supported devices

The SFE Stream Processor supports NVidia Jetson (Xavier, Orin) based devices, for example:


NXP iMX8MPlus installation

Installation instructions for SFE Stream Processor on NXP iMX8M Plus.

Download the latest version of the SFE Stream Processor NXP package at our Customer Portal.

Get the HardwareID and install the license

To generated and deploy the license follow the Licensing instructions.

Configure SFE Stream Processor

Refer to the SFE Stream Processor documentation for more information.

Camera input solver

Before running the SFE Stream Processor application you need to configure the camera_input solver properly to get images from your USB camera.

For more information see Camera input solver and GStreamer input solver parameters.

Run SFE Stream Procesor

Use run.sh to run the SFE Stream Processor with the input from the configured USB camera.

$ ./run.sh

This script also runs the SFE Settings Server application which enables you to configure the SFE Stream Processor via web UI. SFE Settings Server runs by default on localhost:8592.

SmartFace integration

Update the SmartFace Platform to the latest version available at Github. The Linux Docker installation steps are documented here.

To receive notifications from SmartFace Embedded Stream Processor in SmartFace Station you have to configure the SFE Stream Processor connection and create an Edge stream in SmartFace Station Follow the instructions to configure SmartFace Platform.


SFE Stream Processor Client Apps

This page describes how to use the SFE Client application to register users in DB and consume frame_data messages from sfe_stream_processor.

Download the latest version of the SFE Client GUI Application at our Customer Portal. It is available for the following platforms:

  • Windows x86_64
  • Linux x86_64
  • Linux arm64

SFE Client GUI

  1. Run sfe_client_gui executable.
  2. Configure MQTT connection.

  1. Click Connect.
  2. Click Register button under the cropped image of the detected face to register a new user in the database. If the identification is enabled on the camera the face will be matched against all faces in DB and if recognized, the given name will be displayed under the cropped image.

Nx Witness Integration VMS plugin installation

Install Nx Witness Integration VMS and SmartFace Embedded plugin

Follow the instructions to install the required components:

Run and configure the SmartFace Embedded Camera plugin

After you install the DLL, you can enable the SmartFace Embedded Camera plugin in the Camera Settings under Plugins tab

  1. You have to configure the MQTT connection properly. The MQTT configuration (broker, topic and credentials) has to be the same as on the camera running the SFE Stream Processor.
  2. You can choose what information you want to annotate in the camera preview.
  3. You can also configure the face recognition thresholds. These thresholds impact the annotations in the camera preview.
  4. Tolerance in milliseconds for timestamp synchronization between video frames and MQTT frame_data messages.

Please also make sure the camera time is synchronized with the server running Nx Witness Integration VMS.

We also recommend checking the Trust camera timestamp in Camera Settings under Expert tab.

Display events

You need to enable the following events by adding a new event in Camera rules of a specific camera:

  • Analytics Object Event with Face object type to show the face detection and face recognition events

  • Plugin Diagnostics Event to show connection and debug notifications