Skip to main content

Solvers & Platforms

The SmartFace Embedded solvers are binaries processing the NN model inference and model-related pre-processing and post-processing operation. For each NN model, there is a specific solver.

Our NN models can be accelerated on a variety of HW, including CPUs, GPUs and NPUs. Depending on the target platform and inference engine we distinguish different solvers.

Currently, SFE Toolkit supports NN acceleration using the following inference engines.

Note: embedded accelerator targets (NXP i.MX 8, Rockchip, Ambarella) are currently shipped on the SFE Toolkit 3.x line — see the embedded platforms note in the Introduction.

ONNX Runtime​

ONNX Runtime inference engine enables inference of NN models on a variety of HW using different so-called execution providers. ONNX Runtime solver comes with the suffix onnxrt.solver in its name.

ONNX Runtime solvers are currently supported for Windows x86, Linux x86, Jetson ARM64 and Android architectures.

SFE Toolkit currently uses the following ONNX Runtime versions:

  • 1.18 - Linux and Windows (PC)
  • 1.22 - NVidia Jetson with Jetpack 6.x

The ONNX Runtime execution provider can be configured using a solver parameter "runtime_provider" or environment variable ONNXRUNTIME_SOLVER_RUNTIME_PROVIDER. Supported runtime/execution providers are:

  • "cpu" - Linux, Windows, Jetson, Android
  • "cuda" - Linux, Windows, Jetson
  • "tensorrt" - Linux, Windows, Jetson Note: Environment variable overrides solver parameter.

ONNXRuntime solvers depend on the onnxruntime library. For Windows, Microsoft C and C++ (MSVC) runtime libraries are required to be installed see this page.

CPU​

The default ONNX Runtime CPU Execution Provider is MLAS.

The performance and CPU utilization can be configured using the following solver parameters and environment variables:

  • solver parameter "inter_threads" or env variable ONNXRUNTIME_SOLVER_INTER_THREADS
    • Sets the number of threads used to parallelize the execution of the graph (across nodes).
    • Default value is 0 which means the default number of threads will be used.
    • If "parallel" execution mode is turned on, this sets the maximum number of threads to use to run them in parallel.
    • If "sequential" execution mode is enabled this value is ignored, it acts as if it was set to 1.
  • solver parameter "intra_threads" or env variable ONNXRUNTIME_SOLVER_INTRA_THREADS
    • Sets the number of threads used to parallelize the execution within nodes
    • Default value is 0 which means the default number of threads will be used.
  • solver parameter "execution_mode" or env variable ONNXRUNTIME_SOLVER_EXECUTION_MODE
    • Controls whether the operators are executed in parallel or sequentially
    • "parallel" - execute operators in the graph in parallel.
    • "sequential" - execute operators in the graph sequentially.
    • default value is "parallel"
  • solver parameter "log_level" or env variable ONNXRUNTIME_SOLVER_LOG_LEVEL
    • Controls the logging level for ONNX solvers
    • Empty value means no logging
    • Supported values: "verbose", "info", "warning", "error", "fatal"
    • Default value is empty (no logging)
  • solver parameter "log_file" or env variable ONNXRUNTIME_SOLVER_LOG_FILE
    • Specifies the log file path for ONNX solver logging
    • Empty value means logging to stdout
    • Default value is empty (stdout)
  • solver parameter "profile_prefix" or env variable ONNXRUNTIME_SOLVER_PROFILE_PREFIX
    • Enables dumping JSON statistics from inference
    • Specifies the prefix for the profile output files
    • Default value is empty (profiling disabled)
  • solver parameter "session_config" or env variable ONNXRUNTIME_SOLVER_SESSION_CONFIG
    • Supports advanced session settings in ONNX solvers
    • Allows configuration of additional ONNX Runtime session parameters
    • Default value is empty (no additional configuration)

Note: Environment variable overrides solver parameter.

CUDA​

The CUDA Execution Provider enables hardware-accelerated computation on Nvidia CUDA-enabled GPUs and NVidia Jetson platforms

The supported CUDA and cuDNN version requirements are documented in onnxruntime documentation.

You can specify the ID of a CUDA device where the NN model inference will be executed by setting a solver parameter "device_id" or env variable ONNXRUNTIME_SOLVER_DEVICE_ID:

  • default value is 0 Note: Environment variable overrides solver parameter.

TensorRT​

With the TensorRT execution provider, the ONNX Runtime delivers better inferencing performance on the same hardware compared to generic GPU acceleration.

The TensorRT execution provider in the ONNX Runtime makes use of NVIDIA’s TensorRT Deep Learning inferencing engine to accelerate the ONNX model in their family of GPUs.

The supported TensorRT and CUDA versions are documented in onnxruntime documentation

You can configure TensorRT settings by environment variables. Find more details in onnxruntime documentation.

To decrease the ONNX model load time, you can enable the TensorRT engine caching with the following environment variables:

  • ORT_TENSORRT_ENGINE_CACHE_ENABLE: Enable TensorRT engine caching. The default value is 0 (disabled), value 1 means caching is enabled.
  • ORT_TENSORRT_CACHE_PATH: Specify the path for the TensorRT engine and profile files if ORT_TENSORRT_ENGINE_CACHE_ENABLE is 1.

Rockchip NPU​

Rockchip NPU is used for NN model acceleration on Rockchip SoC. SFE Toolkit supports both RKNPU and RKNPU2.

RKNPU​

Supported chipsets are RV1109 and RV1126.

Rockchip solver comes with the suffix rockchip.solver in its name. The solvers are compatible with RKNPU 1.6.0.

RKNPU2​

Supported chipsets are:

  • RK3566/RK3568
  • RK3588/RK3588S
  • RV1103/RV1106
  • RK3562 RV1109 and RV1126.

Rockchip solver comes with the suffix rknn2.solver in its name. The solvers are compatible with RKNPU2 1.5.2.

Ambarella CVFlow​

Ambarella CVFlow is used for NN model acceleration on Ambarella CV28, CV25, CV22 and CV2 chips.

Ambarella solver comes with the suffix ambarella.solver in its name.

We support two different major versions of Ambarella SDK as of today:

  • v3.0
  • v2.5.8.

Solvers built against Ambarella SDK 3.0 use higher-level API (which Ambarella calls EazyAI).

Solvers built against Ambarella SDK v2.5.8 use lower-level API (called NNCTRL).

Tensorflow Lite​

Tensorflow Lite inference engine is used for NN model acceleration on NXP's NPU hardware i.MX 8 series.

Tensorflow Lite solver comes with the suffix tflite.solver in its name.

Acceleration on the NPU is handled by the VX delegate plugin. VX Delegate enables accelerating the inference on on-chip hardware accelerator on i.MX 8 series. The VX Delegate directly uses the hardware accelerator driver (OpenVX with extension) to fully utilize the accelerator capabilities.

SFE Toolkit currently uses Tensorflow Lite 2.9.1.

Supported TFLite solver parameters

  • tflite_solver.delegate
  • tflite_solver.num_threads
  • tflite_solver.vx_delegate.device_id
  • tflite_solver.vx_delegate.cache
  • tflite_solver.vx_delegate.cache_file_path

Supported TFLite solver environment variables

  • TFLITE_SOLVER_DELEGATE
  • TFLITE_SOLVER_NUM_THREADS
  • TFLITE_SOLVER_VX_DELEGATE_DEVICE_ID
  • TFLITE_SOLVER_VX_DELEGATE_ENABLE_CACHING
  • TFLITE_SOLVER_VX_DELEGATE_CACHE_FILE_PATH

Note: Environment variable overrides solver parameter.

Parameters and their values​

Delegate parameter​

Enables users to specify a delegate that should be used for inference. For example, GPU delegate or VX delegate can be specified. Possible values of this parameter, whether it is set via generic solver API or using an environment variable, are:

  • "cpu"
  • "gpu"
  • "nnapi"
  • "vx"

Default value is cpu

Num Threads parameter​

Allows users to specify the number of threads to be used for model inference. Currently, this parameter only affects CPU delegate and does nothing if specified along with a different delegate. Possible values are non-float numbers starting from -1 to infinity`. Special behavior is triggered when the following values are provided:

  • -1 - let the tflite engine choose the most suitable number of threads.
  • 0 - Multithreading disabled.

Default value is -1, which means that the Tensorflow-lite engine will decide how many threads it's going to use.

Vx Delegate device id​

In an environment with multiple VX NPUs, allows you to specify a device ID that a solver should use for inference.

Default value is 0.

Vx Delegate caching enabled and cache file path​

Enables you to turn on the VX model caching. This is useful when the first VX model inference takes too long. It is capable of caching its models into files and reusing them in another instance of the process. You can turn this VX feature on by setting tflite_solver.vx_delegate.cache to "true". By default, it will cache a model into a file with path: /tmp/tflite_solver.vxcache. Default behavior: caching is disabled