logo logo

Data Retention and Backup

Data retention

The SmartFace Platform allows you to set a data retention policy. This also allows you to keep the size of stored data under control.

What data is stored

The SmartFace stores it’s data in two types databases due to different nature of the data stored.

Min.IO

Files and objects are stored in the Min.io database. MinIO is a high-performance, S3 compatible object store. It is suitable for distributed, cloud or on-premises infrastructure. For more information please read project’s webpage.

SQL Databases

The text records and tabular data such as watchlists, watchlist members, templates, match results etc. are stored in SQL databases. The SmartFace supports two database providers:

Microsoft SQL Database

This is the default option for the Windows installation. For more information please read project’s webpage.

Postgre SQL Database

This is the default option for the Docker installation. For more information please read project’s webpage.

Configuring Data Storage In SmartFace

SmartFace Platform can store all data which it recognizes in the incoming live streams or video files. This data may be, for example, whether a face was detected, identified, when it disappeared from a video, on which camera occurred an event and when, etc. All this data can be stored together with cropped images of a person or an object and the full frame from the input.

In various situations, it is required that SmartFace doesn’t store this data. For example, because of the General Data Protection Regulation (GDPR) in European Union, lack of people’s consent for processing of their personal data or simply because of huge requirements on the data storage.

SmartFace Platform allows you to enable or disable the collecting and storing of this data. By disabling data storage, data processed from live streams and video files won’t be stored. Disabling data storage however, doesn’t affect data stored in watchlists.

Data storage options

To view the current configuration and to update the data storage option you can visit the /api/v1/Setup/DataStorage/Video endpoint.

Using the   GET   /api/v1/Setup/DataStorage/Video method you can see your current configuration. Per default it looks like below:

{
  "storageMode": "All"
}

You can enable or disable video data storage by configuring one of the following options in the property storageMode. If you configure the option None, SmartFace Platform stops storing the processed data into the database and stops sending database notifications. Only direct notifications about match results sent from SmartFace Platform will be available to you. For more information, see the chapter Notifications explained.

ℹ️ Please note that face images and templates for watchlist members are still stored in the database.

If you configure the option All, SmartFace Platform stores the events for detected people (regardless if they are registered in a watchlist or not). Data together with the face images and full frames are saved into the database and database notifications are triggered as well. The option All is the default value.

ℹ️ The save strategy All can produce huge amount of data and can cause that you will run out of storage space.

Face Save Strategy

When storageMode is set to All, you should define the configuration FaceSaveStrategy to specify which data will be stored. The following table describes options for FaceSaveStrategy configuration:

Save Strategy OptionDescription
FirstFaceOnly the first detected face of a person is stored.
BestFaceOnly the best face for the detected person is stored. Which face is the best is decided when the person is lost from tracking.
FirstFace and BestFaceThe first and the best face for the detected person are stored.
AllAll faces for the detected person are stored.
MatchedOnlyOnly faces which are matched against a watchlist member are stored.

The default value for FaceSaveStrategy is FirstFace and BestFace.

Full Frame Saving Strategy

It is also possible to configure whether SmartFace stores full frames for a detected person or not. Configuration of this option is in the property SaveFrameImageData with values true or false. The default value is true.

Pedestrian Save Strategy

If a Pedestrian Detector Resource is set for a camera, you can then set the configuration regarding the saving strategy for the pedestrian detector. Depending on your configuration you will either save all the images from a pedestrian tracklet, or only the best image per tracklet. Therefore there are two options: Best and All. The default value is Best.

Object Save Strategy

If an Object Detector Resource is set for a camera, you can then set the configuration regarding the saving strategy for the object detector. Depending on your configuration you will either save all the images from an pedestrian tracklet, or only the best image per tracklet. Therefore there are two options: Best and All. The default value is Best.

Data Retention Preset’s

Please see below the most common presets, with suggested settings for each preset/use case.

unlimited storage capacity solution

  • faceSaveStrategy: all
  • saveFrameImageData: true
  • imageQuality: 100

privacy / GDPR solution

  • faceSaveStrategy: matchedOnly
  • saveFrameImageData: false

compact data solution

  • faceSaveStrategy: FirstFace&BestFace
  • saveFrameImageData: false
  • imageQuality: 70

no data solution

  • videoDataStorageConfig: none

Automatized Database Cleanup

The SmartFace Platform has DB Cleanup functionality. You are able to setup a daily occuring database cleanup. It can remove both images and records - Smartface Platform allows you to manage the deletion of captured data in the databases.

ℹ️ This process will not touch any data related to watchlist/watchlist members

The settings for the DB Cleanup is available in the REST API available on the port 8098 of your installation.

The /api/v1/Setup/DbCleanup endpoint allows two method of interaction:

  1.   GET   /api/v1/Setup/DbCleanup
    Using the GET method you can get the information regarding the current configuration of the DB Cleanup.

The default configuration is as below:

{
  "cleanupAmount": null,
  "enabled": true,
  "maxFramesCount": null,
  "maxImageDataAge": null,
  "cleanupStart": null,
  "deleteSql": false,
  "deleteMatchResults": false
}
  • enabled - defines whether the DB Cleanup is enabled. You can use true and false values
  • cleanupAmount - the amount of image objects to be removed. This value is currently deprecated.
  • maxFramesCount - the maximum number of Frames to keep in database (oldest frames are deleted first). If DeleteSql option is disabled, only blob/image data of these frames will be deleted, SQL data will stay in database.
  • maxImageDataAge - specifies maximum time (in days) to keep frames in database. E.g. when set to 7, all data older than 7 days will be deleted. If DeleteSql option is disabled, only blob/image data of these frames will be deleted, SQL data will stay in database.
  • cleanupStart - CleanupStart defines the time of the day when cleanup should be performed (in UTC). Example value: 01:00:00 which means cleanup will be performed at 1 AM of UTC time.
  • deleteSql - If set to true, cleanup will delete all data related frames from SQL and NoSQL database. If set to false, cleanup will delete only image data of related frames (full frames, crop images of detected faces/pedestrians). SQL data will be persisted.
  • deleteMatchResults - A boolean value indicating whether to also delete match results for faces that are no longer in database.
  1.   PUT   /api/v1/Setup/DbCleanup
    Using the PUT method you can set the new configuration of the DB Cleanup. Please use a JSON format to set the values to match your needs.

Backup and restore data

This section is Docker specific

Docker Volumes

The SmartFace Platform stores it’s data on Docker volumes. The architecture using volumes are useful for backups, restores, and migrations. When the system is installed, 4 docker volumes are automatically created:

  • sf_dependencies_miniodata - stores object and file data, such as images and videos
  • sf_dependencies_mssqldata - stores SQL tabular data for the Microsoft SQL Database
  • sf_dependencies_pgsqldata - stores SQL tabular data for the PostreSQL Database
  • sf_dependencies_rmqdata - stores data for the RabbitMQ message broker

Boths SQL database volumes are created, even though depending on your setup, you will use only one SQL database engine at once. The default SQL database for the Docker version is the PostgreSQL.

Volume Backup

Please follow the steps mentioned below to backup a docker volume. This guide might not solve restoring data backup between all versions. As mentioned below if you are restoring data between two versions you can use the run.sh script for data migration between versions. If in doubt, please contact us for support.

1. Stop the containers

We recommend to stop all docker containers (both dependencies and the main SmartFace set of containers) so there would not be any changes while the backup and/or restore process is ongoing.

You can run the command below in each folder with used docker-compose.yml - both the main and the sf-dependencies folder.

docker-compose stop;docker-compose down

It is possible to stop ALL docker containers at once, but this might interfere with other containers than SmartFace’s, if such containers run on the same server. The command is:

docker stop $(docker ps -a -q)

2. Map an external data folder into a volume

Once the containers are stopped we need to open the docker-compose.yml file in the sf-dependencies folder with your preferred text editor. Please find the definition of the container you would like to backup in the file. We will show the step on a Min.io container. Per default it will look like this:

minio:
    image: "minio/minio:RELEASE.2022-06-02T02-11-04Z"
    container_name: minio
    command: server /data --console-address ":9001"
    ports:
      - "9000:9000"
      - "9001:9001"
    restart: unless-stopped
    volumes:
      - miniodata:/data

Please create a folder you would like to use for a backup of the volume. For the ease of use we can create such folder directly under the sf_dependencies folder. To allow us to backup the minio data we need to add new line under the volumes: section of the Min.io setup, where we can map the backup folder. If the new folder is called backup_minio, the new line would look like this:

- ./backup_minio:/backups

After adding the new like the Min.io container definition will look like this:

minio:
    image: "minio/minio:RELEASE.2022-06-02T02-11-04Z"
    container_name: minio
    command: server /data --console-address ":9001"
    ports:
      - "9000:9000"
      - "9001:9001"
    restart: unless-stopped
    volumes:
      - miniodata:/data
      - ./backup_minio:/backups

In the case of the other containers than the MinIO the paths will be different. For example for the PostreSQL backup mapping it can be ./backups_pgsql:/backups instead.

3. Run the Container

To initiate the mapping we need to run the SmartFace dependencies. In this example we are afer the Min.io. To avoid any changes during the copying, we recommend to keep the SmartFace Platform containers to stay stopped until the whole backup process is completed. You can run the dependencies by going into the sf_dependencies folder and by running the command:

docker-compose up -d

If the docker containers were initiated successfully, the terminal output will look like below:

user@smartface-server:/srv/smartface/sf_dependencies$ docker-compose up -d
Creating pgadmin ... done
Creating mssql   ... done
Creating jaeger  ... done
Creating pgsql   ... done
Creating minio   ... done
Creating rmq     ... done

4. Log into the Container

You can now log into the minio container using command:

docker exec -it minio bash

You will see such output in the terminal with prompt ready to accept commands:

user@smartface-server:/srv/smartface/sf_dependencies$ docker exec -it minio bash
[root@87cf176b4f00 /]#

Now we can copy the content of the data folder into the backups on the minio container.

cp -r /data/* /backups

The reason we copy from the data folder is because the line - miniodata:/data in the docker-compose.yml states so. This means that the Min.io data is mapped to the /data folder inside the Container. The newly created backup folder is mapped as /backups in the Container. If it was for the PostgreSQL container it would be var/lib/postgresql/data/ instead. So the PostgreSQL command would be:

cp -r /var/lib/postgresql/data/* /backups/

Once everything is copied you can actually see the content in the sf-dependencies and it’s backup_minio folders. Please check the content is there to verify the backup process. If no other changes are needed you can run the SmartFace containers again running the below command in the folder where the `docker-compose.yml`` file is located.

docker-compose up -d

Restore volume data

Before trying to restore/update the volume data, please ensure the containers are down, both the main and the dependencies:

docker-compose stop;docker-compose down

Ensure the mapping is set for each container, using steps from the Volume Backup.

Run the containers:

docker-compose up -d

Enter the container (for Min.io as below, for PostreSQL use pgsql instead of minio)

docker exec -it minio bash

Copy and replace the data from the backups into the data of the container. Ensure that only the backed up data are in the new folder. Do not mix old and new data.

cp -r /backups/* /data/

It is possible that such operation won’t be allowed by the container as the container is up and running and can occupy/use a file/folder that you would like to override/restore into. In such case you need to setup the container the way that it is up and running however it is not yet running for it’s main purpose - the functionality and services inside are not yet available. You can do so by altering the entrypoint temporarily. This change needs to be removed once the restoration is completed.

To adjust the entrypoint you need to add the line below to the container setup in the `docker-compose.yml`` file:

entrypoint: sleep 1h

The Min.io container definition will look like this:

minio:
    image: "minio/minio:RELEASE.2022-06-02T02-11-04Z"
    container_name: minio
    command: server /data --console-address ":9001"
    ports:
      - "9000:9000"
      - "9001:9001"
    restart: unless-stopped
    volumes:
      - miniodata:/data
      - ./backup_minio:/backups
    entrypoint: sleep 1h

The value for the sleep command is by default in seconds, i.e. sleep 100 equals to 100 seconds. However you can use units too, where sleep 1h will equal to 1 hour etc.

Once it is set run all the containers by command in the main docker-compose.yml folder

docker-compose up -d

If the versions between the backup and restore do not match - you are restoring data to a newer version you need to run the run.sh script that will migrate the data. Script is located with the main SmartFace docker-compose.yml file in the same directory.

run.sh