DOT Web Face Auto Capture
v4.0.3
Introduction
DOT Web Face Auto Capture is set of Non-ui component and Ui component web components to capture image of user’s face.
Non-ui component is a web component that renders the video stream from an available phone or web camera to automatically capture an image of a user’s face with the required quality.
Ui component is a web component that renders an overlay over video stream. Overlay includes a placeholder, camera control buttons and instructions to guide the user to position their face correctly.
Supported Browsers
DOT Web Face Auto Capture was tested with:
Chrome on desktop (Windows, Mac and Linux) and mobile (Android, iPhone)
Firefox on desktop (Windows, Mac and Linux) and mobile (Android)
Edge on Windows
Safari on Mac, iPad and iPhone
WebView on Android*
SafariVC on iPad and iPhone**
Known issues:
Components don’t work with WKWebView on iPhone
* Some older Android phones can experience issues with camera stream initialization using WebRTC. Our web components will work on most of them, but camera stream could be slow and laggy. This issue is caused by device and can be experienced in any website using WebRTC in Android WebView.
** Components are tested with SafariVC on devices iPhone 7 and newer with iOS 15 or iPadOS 15 or newer. Older devices or iOS versions are not officially supported and might experience issues.
Privacy and security
This component can only be used in secure contexts due to MediaDevices
API used for handling camera access. A secure context is, in short, a page loaded using HTTPS or the file:///
URL scheme, or a page loaded from localhost
. Before accessing any camera, the component must always get the user’s permission. Browsers may offer a once-per-domain permission feature, but they must ask at least the first time, and the user has to specifically grant ongoing permission if they choose to do so. Browsers are required to display an indicator that shows that a camera or microphone is in use. More details can be found on MDN docs.
Non-ui component
Requirements
Minimum required camera resolution for appropriate results is 720p. Anything less than 720p is insufficient.
Initialization
DOT Web Face Auto Capture can be installed via NPM, yarn or pnpm
npm install @innovatrics/dot-face-auto-capture
To manually integrate the DOT Web Face Auto Capture, download latest version from the Github repository. Then add following line to dependencies in your package.json:
"dependencies": {
"@innovatrics/dot-face-auto-capture": "file:dot-face-auto-capture-[VERSION].tgz"
},
where [VERSION]
is the DOT Web Face Auto Capture version integrated. This installs dot-face-auto-capture
as an external module that can then be used (just like any other module in the code). For example, one could do import '@innovatrics/dot-face-auto-capture';
in the app.
Usage
Face auto capture component is a web component which uses custom HTML <x-dot-face-auto-capture/>
tag.
Properties cameraOptions
needs to be passed into component after <x-dot-face-auto-capture/>
tag was rendered.
import type {
FaceCameraProps,
FaceComponentData,
HTMLFaceCaptureElement
} from "@innovatrics/dot-face-auto-capture";
import { useEffect, useCallback } from "react";
import "@innovatrics/dot-face-auto-capture";
const FaceCamera = (props: FaceCameraProps) => {
useEffect(() => {
const faceAutoCaptureHTMLElement = document.getElementById(
"x-dot-face-auto-capture"
) as HTMLFaceCaptureElement | null;
if (faceAutoCaptureHTMLElement) {
faceAutoCaptureHTMLElement.cameraOptions = props;
}
});
return <x-dot-face-auto-capture id="x-dot-face-auto-capture" />;
};
const Page = () => {
const handleFacePhotoTaken = (image: Blob, resolution: FaceComponentData) => {
// ...
};
// Save function reference to prevent unnecessary reload of component
const handleError = useCallback(
(error: Error) => {
alert(error)
},
[],
);
return (
<FaceCamera
imageType="png"
cameraFacing="user"
photoTakenCb={handleFacePhotoTaken}
onError={handleError}
/>
);
};
See also DOT Web Samples showing the usage of DOT Web Auto Capture components in different front-end technologies like React, Angular…
TypeScript
Declaration file is bundled inside package. To use with TypeScript, import types from @innovatrics/dot-face-auto-capture
.
import type { FaceCallback, FaceCameraProps } from "@innovatrics/dot-face-auto-capture";
Hosting SAM wasm
The component needs to have access to the WebAssembly wasm binary file. It’s distributed in the package, and needs to be hosted by the website provider.
By default, component try to fetch wasm file from <PROJECT_ORIGIN>/sam.wasm
. This can be changed using the samWasmUrl
property.
If using Create React App
, copy the`sam.wasm` file to public
folder. In our example the final path is public/sam.wasm
.
Properties
(Optional)
['png']
string imageType
– Format of the image returned after successful capture'jpeg'
'png'
(Optional)
['user']
string cameraFacing
– Defines which camera to acquire from the browser’s getUserMedia API'user'
– The video source is facing toward the user; this is the selfie or front-facing camera on a smartphone'environment'
– The video source is facing away from the user, thereby viewing their environment; this is the back camera on a smartphone
function onPhotoTaken
– Callback on successful image capture(image, data) ⇒ void
- (see Callback parameters)
function onError
– Callback in the case that an error occurred (see Handling errors)(e: Error) ⇒ void
(Optional)
string samWasmUrl
- URL link to the location where the wasm binary file is hosted(Optional)
object thresholds
- Detection configuration(Optional)
[0.12]
number faceConfidence
- Detection confidence threshold(Optional)
[400]
number sharpnessThreshold
- Low sharpness threshold(Optional)
[400]
number brightnessLowThreshold
- Low brightness threshold(Optional)
[900]
number brightnessHighThreshold
- High brightness threshold(optional)
[0.05]
number outOfBoundsThreshold
- Face out of bounds threshold(Optional)
[0.35]
number minFaceSizeRatio
- Minimum face size ratio relative to shorter side of the camera resolution(Optional)
[0.44]
number maxFaceSizeRatio
- Maximum face size ratio relative to shorter side of the camera resolution
Callback parameters
Blob image
– Returned image on successful captureobject data
object cameraSettings
- MediaTrackSettings object containing used webcam settings. The object in addition contains device name.(Optional) object detection
- Object contains all detection parameters and its values. Present if image was taken using auto capture (not manual capture).object imageResolution
- Width and height of the captured image.
Multi capture
Face auto capture component allows you to capture an unlimited number of faces without the need to reinitialize the webcam and detector.
Component calls photoTakenCb
callback on every captured face photo. When photoTakenCb
is called, detection is paused. Camera stream and face detector stay initialized.
Component is in waiting state. You should implement a custom UI for waiting state (e.g. overlay over our component). We provide a default UI, but it is not customizable.
Implement dispatch 'continue-detection' event
import { dispatchControlEvent, FaceCustomEvent, ControlEventInstruction } from '@innovatrics/dot-face-auto-capture/events';
const continueDetection = () => {
dispatchControlEvent(FaceCustomEvent.CONTROL, ControlEventInstruction.CONTINUE_DETECTION)
};
Without importing @innovatrics/dot-face-auto-capture/events
.
function continueDetection() {
document.dispatchEvent(
new CustomEvent('face-auto-capture:control', {
detail: { instruction: 'continue-detection' },
}),
);
}
Handling errors
When an error occurs, we call onError
callback with one parameter of type Error
. We set name
property to AutoCaptureError
, and also message
with more details.
Component renders default UI for error state, but is not customizable, and integrator should implement own error handling.
Component uses the MediaDevices API that provides access to connected media input devices like cameras and microphones.
If the user denies permission to access or the webcam is not present, an error is thrown. We provide original error thrown by browser inside cause
property of the error object.
List of possible errors can be found on MDN docs.
Error example:
{
name: "AutoCaptureError",
message: "The webcam is already in use by another application",
cause: DOMException: Could not start video source // Original error thrown by browser MediaDevices API
}
Ui component
Requirements
Both components must be wrapped in parent div with position: relative
.
<div style={{position: 'relative'}}>
<FaceCamera
imageType="png"
cameraFacing="user"
onPhotoTaken={handlePhotoTaken}
onError={onError}
/>
<FaceUi />
</div>
Initialization
UI component can be installed via NPM, yarn or pnpm
npm install @innovatrics/dot-auto-capture-ui
To manually integrate UI component, download latest version from the Github repository. Add following line to dependencies in your package.json:
"dependencies": {
"@innovatrics/dot-auto-capture-ui": "file:dot-auto-capture-ui-[VERSION].tgz",
}
where [VERSION]
is the DOT Web Face Auto Capture version integrated. This installs dot-auto-capture-ui
as an external module that can be used (just like any other module in the code). For example, one could do import '@innovatrics/dot-auto-capture-ui';
in the app.
Usage
Face auto capture UI component is an web component which uses custom HTML <x-dot-face-auto-capture-ui />
tag.
Properties props
needs to be passed into component after <x-dot-face-auto-capture-ui />
tag was rendered.
import type { FaceUiProps, HTMLFacetUiElement } from '@innovatrics/dot-autocapture-ui/face';
import '@innovatrics/dot-auto-capture-ui/face';
import { useEffect } from 'react';
const FaceUi = (props: FaceUiProps) => {
useEffect(() => {
const uiElement = document.getElementById('x-dot-face-auto-capture-ui') as HTMLFacetUiElement | null;
if (uiElement) {
uiElement.props = props;
}
});
return <x-dot-face-auto-capture-ui id="x-dot-face-auto-capture-ui" />;
};
TypeScript
Declaration files are bundled inside package. Just import types from @innovatrics/dot-auto-capture-ui/face
.
Properties
(Optional)
enum placeholder
- One of the predefined placeholders in component that can be selectedCIRCLE_SOLID = 'circle-solid'
ELLIPSE_SOLID = 'ellipse-solid'
MAN_SOLID = 'man-solid'
WOMAN_SOLID = 'woman-solid'
SQUARE_ROUNDED_DASH = 'square-rounded-dash'
SQUARE_ROUNDED_SOLID = 'square-rounded-solid'
SQUARE_DASH = 'square-dash'
SQUARE_SOLID = 'square-solid'
(Optional)
object instructions
- Modification of default messages for localization or customization(Optional)
['Stay still…']
string candidate_selection
- Shown when all validations are passed, i.e. the image is suitable for capture(Optional)
['Move back']
string face_too_close
- Shown when the face is too close to the camera(Optional)
['Move closer']
string face_too_far
- Shown when the face is too far from the camera(Optional)
['Center your face']
string face_centering
- Shown when the face is not centered in the placeholder(Optional)
['Position your face into the circle']
string face_not_present
- Shown when no face is detected(Optional)
['Turn face against light']
string sharpness_too_low
- Shown when the face in the image is not sharp enough(Optional)
['Turn face against light']
string brightness_too_low
- Shown when the image is too dark(Optional)
['Less light needed']
string brightness_too_high
- Shown when the image is too bright.
(Optional)
theme
- Modification of theme properties.(Optional)
colors
- To customize colors of following properties(Optional)
['white']
color placeholderColor
- Color of the placeholder lines(Optional)
['#00BFB2']
color placeholderColorSuccess
- Color of the placeholder lines when all validations are passed(Optional)
['white']
color instructionColor
- Instruction background color(Optional)
['#00BFB2']
color instructionColorSuccess
- Instruction background color when all validations are passed(Optional)
['black']
color instructionTextColor
- Instruction text color
(Optional)
appStateInstructions
- Modification of default messages for component state(Optional)
loading
- Component loading state(Optional)
['Loading. Please wait.']
string text
- Text shown while component is loading(Optional)
[true]
boolean visible
- Show/hide loading instruction while component is loading
(Optional)
waiting
- Component waiting state(Optional)
['Waiting for input']
string text
- Text shown while component is waiting(Optional)
[true]
boolean visible
- Show/hide waiting instruction while component is waiting
(Optional)
[false]
boolean
showDetectionLayer
- Show/hide detection layer on top of component(Optional)
[false]
boolean
showCameraButtons
- Show/hide camera buttons (switch camera and toggle mirror) on top of component
Example how to use UI properties:
import { FacePlaceholderIcon } from '@innovatrics/dot-auto-capture-ui/face';
<FaceUi
theme={{ colors: { placeholderColor: 'green' } }}
placeholder={FacePlaceholderIcon.ELLIPSE_SOLID}
instructions={{ candidate_selection: 'Candidate selection' }}
appStateInstructions={{ loading: { text: 'Component is loading', visible: true } }}
showDetectionLayer
showCameraButtons
/>
In javascript you can use value of FacePlaceholderIcon
directly.
<FaceUi placeholder="ellipse-solid" />
Custom Events - Optional
Communication between components is based on custom events. Ui
component is mostly listening to events dispatched by non-ui
component. In case of CONTROL
event, ui
component dispatches control events to control non-ui
component.
When using default [face-auto-capture-ui] component, only dispatch continue-detection
event has to be implemented to use Multi capture.
Type of Events
Currently all events being used are described in this section. These are sufficient to build custom UI layer.
enum FaceCustomEvent
- Event names dispatched by componentsCONTROL = 'face-auto-capture:control
- Events dispatched from 'ui' component to controlnon-ui
component. Described in Control Events section
CAMERA_PROPS_CHANGED = 'face-auto-capture:camera-props-changed'
- Notifies UI when camera properties has changedINSTRUCTION_CHANGED = 'face-auto-capture:instruction-changed'
- Notifies UI when instruction has changedSTATE_CHANGED = 'face-auto-capture:state-changed'
- Notifies UI when state ofnon-ui
component has changedDETECTED_FACE_CHANGED = 'face-auto-capture:detected-face-changed'
- Notifies UI when face is detected. Used in Detection Layer, has no effect ifshowDetectionLayer = false
VIDEO_ELEMENT_SIZE = 'face-auto-capture:video-element-size'
- Notifies UI when HTML video element size has changed
Usage
Import @innovatrics/dot-face-auto-capture/events
to use event types and and dispatch control events.
Listening to the Events
All event listeners are already implemented in default [face-auto-capture-ui] component. Skip this section if you are using default ui
component.
All FaceCustomEvent
events, except CONTROL
, are dispatched by Face Auto Capture non-ui
component. Ui
component listens to these events to make appropriate changes. See the example below how to register event listeners.
import type {
CameraPropsChangeEvent,
CameraStateChangeEvent,
VideoElementSizeChangeEvent,
FaceInstructionChangeEvent
DetectedFaceChangeEvent
} from '@innovatrics/dot-face-auto-capture/events'
import { FaceCustomEvent } from '@innovatrics/dot-face-auto-capture/events';
import type {
AppState,
FaceInstructionCode,
Resolution,
AutoCaptureError
} from '@innovatrics/dot-face-auto-capture';
import { useEffect, useState } from 'react'
export const Events = () => {
const [instructionCode, setInstructionCode] = useState<FaceInstructionCode | undefined>();
const [cameraResolution, setCameraResolution] = useState<Resolution | undefined>();
const [isMirroring, setIsMirroring] = useState<boolean | undefined>();
const [appState, setAppState] = useState<AppState | undefined>();
const [error, setError] = useState<AutoCaptureError | undefined>();
const [videoElementSize, setVideoElementSize] = useState<DOMRect | undefined>();
const [detectedFaceCorners, setDetectedFaceCorners] = useState<DetectedFaceCorners | undefined>();
useEffect(() => {
const handleInstruction = (event: FaceInstructionChangeEvent) => {
setInstructionCode(event?.detail?.instructionCode);
};
const handleCameraProps = (event: CameraPropsChangeEvent) => {
setCameraResolution(event?.detail?.cameraResolution);
setIsMirroring(event?.detail?.isMirroring);
};
const handleAppState = (event: CameraStateChangeEvent) => {
setAppState(event?.detail?.appState);
const error = event?.detail?.error;
if (error) {
setError(error);
}
};
const handleVideoElementSize = (event: VideoElementSizeChangeEvent) => {
setVideoElementSize(event.detail?.size)
}
const handleDetectedFace = (event: DetectedFaceChangeEvent) => {
setDetectedFaceCorners(event?.detail?.detectedCorners)
}
document.addEventListener(FaceCustomEvent.INSTRUCTION_CHANGED, handleInstruction);
document.addEventListener(FaceCustomEvent.CAMERA_PROPS_CHANGED, handleCameraProps);
document.addEventListener(FaceCustomEvent.STATE_CHANGED, handleAppState);
document.addEventListener(FaceCustomEvent.VIDEO_ELEMENT_SIZE, handleVideoElementSize)
document.addEventListener(FaceCustomEvent.DETECTED_FACE_CHANGED, handleDetectedFace)
return () => {
document.removeEventListener(FaceCustomEvent.INSTRUCTION_CHANGED, handleInstruction);
document.removeEventListener(FaceCustomEvent.CAMERA_PROPS_CHANGED, handleCameraProps);
document.removeEventListener(FaceCustomEvent.STATE_CHANGED, handleAppState);
document.removeEventListener(FaceCustomEvent.VIDEO_ELEMENT_SIZE, handleVideoElementSize);
document.removeEventListener(FaceCustomEvent.DETECTED_FACE_CHANGED, handleDetectedFace)
};
}, [])
}
Without importing @innovatrics/dot-face-auto-capture/events
you can use values of FaceCustomEvent
directly.
const instructionChangeEvent = "face-auto-capture:instruction-changed";
function handleInstructionChange(event) {
console.log(event.detail.instructionCode)
}
document.addEventListener(instructionChangeEvent, handleInstructionChange);
/**
* remove event listener when you're done
*/
document.removeEventListener(instructionChangeEvent, handleInstructionChange);
Control Events
Control events are dispatched from ui
component to control non-ui
component.
enum ControlEventInstruction
CONTINUE_DETECTION = 'continue-detection'
- Controls Multi captureSWITCH_CAMERA = 'switch-camera'
- Notifies Face Auto Capture to use different cameraTOGGLE_MIRROR = 'toggle-mirror'
- Notifies Face Auto Capture to mirror video stream
import { dispatchControlEvent, FaceCustomEvent, ControlEventInstruction } from '@innovatrics/dot-face-auto-capture/events';
export const Dispatch = () => {
const continueDetection = () => {
dispatchControlEvent(FaceCustomEvent.CONTROL, ControlEventInstruction.CONTINUE_DETECTION)
}
const switchCamera = () => {
dispatchControlEvent(FaceCustomEvent.CONTROL, ControlEventInstruction.SWITCH_CAMERA)
}
const toggleMirror = () => {
dispatchControlEvent(FaceCustomEvent.CONTROL, ControlEventInstruction.TOGGLE_MIRROR)
}
return (
<div>
<button onClick={continueDetection}>Continue detection</button>
<button onClick={switchCamera}>Switch camera</button>
<button onClick={toggleMirror}>Mirror camera</button>
</div>
)
}
Without importing @innovatrics/dot-face-auto-capture/events
you can use values of FaceCustomEvent
and ControlEventInstruction
directly.
const continueDetection = () => {
document.dispatchEvent(
new CustomEvent('face-auto-capture:control', {
detail: { instruction: 'continue-detection' },
}),
);
}
Example of using components together
import type { FaceComponentData } from '@innovatrics/dot-face-auto-capture';
import {
dispatchControlEvent,
FaceCustomEvent,
ControlEventInstruction
} from '@innovatrics/dot-face-auto-capture/events'
import { useCallback } from 'react';
import FaceCamera from './FaceCamera'
import FaceUi from './FaceUi'
const FaceAutoCapture = () => {
const handlePhotoTaken = (image: Blob, data: FaceComponentData) => {
console.log(image, data);
}
const handleError = useCallback(
(error: Error) => {
alert(error)
},
[],
);
const handleContinueDetection = () => {
dispatchControlEvent(FaceCustomEvent.CONTROL, ControlEventInstruction.CONTINUE_DETECTION)
};
return (
<div>
<button onClick={handleContinueDetection}>Continue detection</button>
<div style={{ position: 'relative' }}>
<FaceCamera
imageType="png"
cameraFacing="user"
onPhotoTaken={handlePhotoTaken}
onError={handleError}
/>
<FaceUi />
</div>
</div>
);
};
Appendix
Changelog
Fixed
Detection not initialized on slow internet connection. Component remains in "waiting" state
"Candidate selection phase" at the beginning of detection when detecting multiple captures
applied
text-align: center
on instruction component on small screen
Changed
changed instruction when face_not_present
4.0.1 - 2023-01-19
Fixed
export events from
@innovatrics/dot-face-auto-capture/events
4.0.0 - 2023-01-18
Removed
UI layer from camera component (buttons, placeholder, instruction, app state overlay and detection layer)
center face validator
FACE_CENTERING_NARROW_SIDE
thresholdFACE_CENTERING_WIDE_SIDE
threshold
Added
UI package with placeholder, instructions, buttons and detection layer
Custom events and events listeners for communication from app to component (switch-camera, toggle-mirror)
Custom events and events listeners for communication from components to app (camera-resolution-changed, instruction-changed, detected-document-changed, state-changed)
detection layer with detected face circle
showDetectionLayer
switch for show/hide camera buttons (switch camera, toggle mirror)showCameraButtons
switch for show/hide detection layerOutOfBounds
validatoroutOfBoundsValidationRectangle
outOfBoundsDetectionPoints
all rectangles and points into debug layer
SAM version
into debug layer
Changed
Rename
photoTakenCb
toonPhotoTaken
Order of instruction processing. Before validating the centering of the detected face, the size of the detected face is validated.
Component is now build as
umd
andes module
Camera buttons (switch camera, toggle mirror) are now hidden by default
3.5.2 - 2022-12-16
Fixed
Camera switch on Android devices with Firefox
3.5.1 - 2022-10-13
Fixed
Memoized props from first instance which holds reference to callback function from initialization on every capture
3.5.0 - 2022-10-10
Changed
Best candidate selection by sharpness
Dynamic candidate selection phase duration
Center face narrow side validator from 0.07 to 0.12
Fixed
Minimal camera resolution check
Selection of best image in candidate selection phase entering
Removed
Hotspots validator with
hotspotsMediumThreshold
andnumber hotspotsHighThreshold
3.4.3 - 2022-09-28
Fixed
Possible false candidate selection on first frame after entering candidate selection
3.4.2 - 2022-08-10
Fixed
Typescript declaration files export
3.3.1 - 2022-07-04
Fixed
Zero height of component when error occurs
Add
add
appStateInstructions
property to uiCustomisationadd option to change appState instructions text and visibility
Change
interface for changing
Loading
appState instruction text
3.3.0 - 2022-06-03
Added
Add support for capturing multiple face photos
Add custom event
face-auto-capture
withcontinue-detection
instruction to continue detection
Changed
When
photoTakenCb
is called, component switches into waiting stateParent of
<x-dot-face-auto-capture />
does not have to have a defined height
3.2.0 - 2022-05-23
Added
Add
Countly
analytics trackingAdd image parameters (brightness, sharpness, hotspot)
Session recording for face detection
Changed
onError
callback returnsAutoCaptureError
onError
callback is requiredShow error screen when error occurs
Fixed
Stucked component when error occurs
3.1.2 - 2022-04-04
Changed
Remove force manual capture on instruction click
3.1.1 - 2022-03-15
Changed
Change
optimalFaceSizeLimit
andoptimalFaceSizeParam
tominFaceSizeRatio
andmaxFaceSizeRatio
Change
faceCenterLimit
tofaceCenteringNarrowSide
andfaceCenteringWideSide
Set
minFaceSizeRatio
default value to0.35
andmaxFaceSizeRatio
to0.44
Set
faceCenteringNarrowSide
default value to0.07
andfaceCenteringWideSide
to0.21
3.1.0 - 2022-02-03
Fixed
Unify loading and instruction design
3.0.2 - 2022-01-28
Added
Add loading screen
Add
loading.text
property inuiCustomisation
for configuring text on loading screen
Changed
Show new instruction only if 600ms elapsed since last instruction change
3.0.1 - 2022-01-25
Added
file
sam.wasm
Fixed
safari version 15 on mac
Changed
Face detection accuracy improved.
Detection engine
Removed
blaze_models
packagemodelUrls
option from cameraOptions
2.3.1 - 2022-01-10
Fixed
Fix bug causing integration error
2.3.0 - 2022-01-05
Added
add
cameraSettings
intophotoTakenCb
callBack functionRedesign of the UI of Face Auto Capture component.
Changed
photoTakenCb
callBack function structure
2.2.2 - 2021-12-17
Incremental version upgrade
2.2.1 - 2021-12-09
Changed
Hide switch camera button if there is only 1 webcam available
2.2.0 - 2021-11-30
Added
thresholds.faceConfidence
- Detection confidence threshold with default value0.12
modelUrls
- Object with path to the models needed for detection
Changed
integrated new neural network for face detection
dynamic import of TensorFlow.js libraries
2.1.0 - 2021-11-18
First released version