ID Document Data Extraction - OCR

Optical Character Recognition (OCR) functionality of the Digital Identity Service (DIS) processes the image of identity documents of customer whose identity is verified, and extracts text and image data from those images.

The recognized text data is returned as an object that describes the customer that owns the document. The data are split between those relevant to the “customer” himself and those relevant to the “document” of that customer. The data structure is described in the response of the GET customers API call. Each standard ID document should contain following data:

Each text field is assigned to a category that is universal across all documents, such as “surname”. It is possible to also retrieve the label - name of that text field as stated on the document using the GET metadata API call.

Some text fields, such as dates or gender, are normalized to standard format, so that it is possible to operate with documents of multiple countries using the same logic.

If there is an MRZ field, it is parsed and its information are returned as an alternative value of the text fields. The MRZ field is also used for cross-checking the data from the Visual Inspection Zone (VIZ).

DIS also returns image fields, such as primary face photo, images of signature or fingerprints.

API to request OCR#

To perform document OCR, at first the customer and customer’s document entities needs to be created. This is done with the POST customers API call and PUT document API call.

After that, the photos of the ID document pages have to be uploaded - front and back side for ID cards, or just a single page for passport, using the PUT document/pages API call.

The recognized data are retrieved with the GET customers API call.

Definition of Identity Documents supported by Innovatrics#

Identity document is an official document that proves person’s identity. This is done by providing person’s face and the name and other details that uniquely identify then.

Data needed to uniquely identify a person:

• face photo
• name (full name whether in one field or split to multiple fields)
• date of the birth
• additional identifiers needed, like personal number, parent’s names, etc.

Data needed for unique ID document:

• document unique number
• date of document expiry (There are existing documents that do not expire, but this practice is now avoided by governments, as the photos need to be updated on the documents regularly to cope with person’s aging. Old documents without expiration can be supported, but may have downsides.)

Dimensions of supported documents:

• TD1 86 x 54 mm
• TD2 105 x 74 mm
• TD3 125 x 88 mm
• non-standardized documents with similar dimensions

Supported Identity Document Types#

DIS can support identity documents of the following types:

• Passports
• Identity cards
• Driving licenses
• Foreigner permanent residence cards
• and other cards of similar format containing a photo of the holder

The support for document recognition is in multiple levels:

• NOT_SUPPORTED is for documents where DIS models do not support reading of any text. The document page is searched in such case for a face portrait that can be used for matching with selfie.
• MRZ_EXTRACTION_ONLY, also as Level 1 means that document does not have a specific template, the extraction is done from the data found in the MRZ zone.
• GENERIC_SUPPORT is reported for US and Canadian driving licences and identification cards whose text is read by the generic document reader - one model that reads the visual zone of documents without a dedicated template. Passports and EU/EEA identity cards read by the same reader without a template keep their earlier values: FULL_SUPPORT with supportMethod: GLOBAL_PASSPORT_SUPPORT for passports and GLOBAL_DOCUMENT_SUPPORT for identity cards. The reader is enabled by default and can be switched off with innovatrics.dot.dis.customer.document.global-ocr-passport-detection.enabled: false.
• FULL_SUPPORT, also as Level 2 means that the document has a dedicated template and extraction of all the data is done in both visual inspection zone (VIZ) and MRZ zone (if present).

For Full Level 2 support, DIS needs to be trained to support each individual document type and its edition. Please check the availability of the required document in the list of supported documents for both levels. In the case an ID document type required for your project is not mentioned in the list, a future version of DIS can be trained to support it. Please contact Innovatrics to request it.

Document Support Levels#

The document falls into the support level based on these conditions:

Generic OCR Support for Global Passports#

If a passport is not fully supported, a separate model - the generic document reader - is used to read its visual zone. This one detects the texts in the document and identifies their field type. The greatest benefit of this model compared to MRZ extraction is that the names are not truncated in the visual zone. The same reader also reads US and Canadian driving licences and identification cards, with or without a trained template. The following Latin text fields are supported by this model:

• Full name, or Given names and Surname
• Document number
• Personal number
• Date of Birth
• Date of Expiry
• Date of Issue
• Nationality
• Place of birth
• Issuing Authority

US and Canadian driving licences and identification cards#

From the next DIS release, driving licences and identification cards issued by the US states and territories and by the Canadian provinces - the AAMVA jurisdictions - are read by the generic document reader, whether or not DIS has a trained template for them. The reader needs to be enabled (it is by default, see above).

Cards with a trained template (see the list of supported documents) are classified by the best-matching template and reported with that template’s documentType (country, type and edition); the template’s own validation of its read no longer decides the classification. Their text, front and back page alike, is read by the generic document reader instead of the template. The template read runs alongside: it supplies the portrait, signature and barcode crops and the authenticity scores, and it is the fallback when the generic reader returns nothing usable (a front needs the document number and a name, a back any field with a value). A page whose template reads an MRZ keeps the template read. Such a page is reported with supportLevel: GENERIC_SUPPORT and supportMethod: GLOBAL_DOCUMENT_SUPPORT instead of FULL_SUPPORT. Passports, work and residence permits, and the documents of every other country keep the template read. The order is configurable with innovatrics.dot.dis.customer.document.global-ocr.preference.aamva (GENERIC_FIRST, the default, or TEMPLATE_FIRST, which restores the template read and FULL_SUPPORT).

What GENERIC_SUPPORT means for the integrator on a US or Canadian page:

• The reader does not measure OCR confidence. visualZoneInspection.ocrConfidence is reported as 1.0 and no low-confidence texts are reported, so a “review this field” decision cannot be based on it.
• The template’s colour check still runs on a page that a template classifies, so colorProfileChangeDetected is reported; it is absent on pages without a template. tamperedTexts is absent: the templates of these cards carry no text authenticity model. See Document Authenticity.
• The visual-zone texts may differ from what the template would have read.

Cards without a trained template are classified from the PDF417 barcode on their back, which carries the card data in the AAMVA format. Only the barcode classifies such a page; no classification is made from its text alone.

• Decoded barcode: the page is reported as the back of a document with documentType.country USA or CAN - decided by the barcode’s country element DCG - and type drivers-licence or identity-card, decided by the barcode’s DL or ID subfile. A non-driver AAMVA identification card is therefore an identity-card, not a driving licence. edition carries the issuing state or province as a lowercase two-letter code (e.g. nj, on), derived from the barcode’s issuer identification number, and is absent when the issuer is not known; Newfoundland and Labrador is nl, not the NF the AAMVA issuer list prints. machineReadableTravelDocument is AAMVA and, until a front page is read, supportLevel is MRZ_EXTRACTION_ONLY. A decoded AAMVA barcode of any other country, or a decoded barcode of another format, does not classify the page; neither does an AAMVA barcode without the DCG element - such a card predates the 2005 AAMVA standard and has expired. When the BARCODE document source is requested, GET customers returns the decoded barcode in document.barcodes and its AAMVA fields as the barcode value of the matching attributes: givenNames (first and middle name joined), surname, dateOfBirth, document.documentNumber, document.dateOfIssue, document.dateOfExpiry, and the document.additionalTexts height, eyesColor, hair, weight, class, restrictions, endorsements, address and securityNumber, with dates in YYYY-MM-DD. This applies to every document whose AAMVA barcode is decoded, trained templates included; gender has no barcode value. The same fields are cross-checked with the visual zone by document/inspect.
• Barcode located but not decodable (damaged, glare, low resolution): the page is still classified as the back of a US driving licence - documentType.country: USA, type: drivers-licence, no edition, no barcode data - and its text is read by the generic document reader (supportLevel: GENERIC_SUPPORT, supportMethod: GLOBAL_DOCUMENT_SUPPORT). When that read yields nothing, the classification stands alone with supportLevel: MRZ_EXTRACTION_ONLY.

Note: No country can be read from a barcode that cannot be decoded, so the classification above is an accepted false positive: a Canadian card, or any other card carrying a PDF417 that DIS cannot decode, is reported the same way. Treat documentType.country: USA on a page without edition and without barcode data as a hint, not a verified fact, and validate it in your integration - against the text the generic reader returned, by asking the user to confirm the issuing country, or by a recapture.

Front page of a card without a template. A page that no template, MRZ or barcode classifies, and on which a portrait is detected, becomes the document’s front once a barcode-classified US or Canadian back of the same customer is present - in either upload order. Its text is read by the generic document reader with the back’s country (a Canadian back confirms a Canadian front). The document then reports pageTypes: [front, back], the back’s documentType (country, type, edition) and supportLevel: GENERIC_SUPPORT with supportMethod: GLOBAL_DOCUMENT_SUPPORT. A portrait page on which the reader finds no document number and name is not paired: it stays unknown and the back reports the MISSING_FRONT_PAGE warning, so a paired document always carries a visual zone the barcode can be checked against. A portrait page with no such back, or uploaded when the document already holds a front, is reported as before: DOCUMENT_TYPE_NOT_RECOGNIZED, page side unknown. A back of any other document never confirms it. The upload that completes the pair takes one generic read longer, because the front is read at that point.

Front-only capture on advice. When the document is created (PUT document) with classification advice that names exactly one AAMVA jurisdiction - advice.classification.countries: ["USA"] or ["CAN"], or a US territory (PRI, GUM, VIR, ASM, MNP), optionally with types: ["drivers-licence"] or ["identity-card"] - a page that no template, MRZ or barcode classifies is read by the generic document reader with that country. When the read yields a document number and a name, and a portrait is detected on the page, the page is reported as the front: documentType.country and type from the advice (drivers-licence when no type was advised), machineReadableTravelDocument: AAMVA, supportLevel: GENERIC_SUPPORT, supportMethod: GLOBAL_DOCUMENT_SUPPORT. The customer’s document is complete with that page alone - no back page or barcode is required - while document/inspect still derives the expected pages from the card’s format and lists the back as missing. Without a portrait, or when the reader finds no document number and name, the page stays DOCUMENT_TYPE_NOT_RECOGNIZED with page side unknown. An advised US territory is reported the way its barcode would report it: documentType.country: USA with the territory as edition (pr, gu, vi, as, mp); advice for the USA or Canada gives no edition. The advice is a hint, not proof: a page that a trained template of another country recognizes is not read as the advised document and stays unknown (DOCUMENT_TYPE_NOT_RECOGNIZED). Such advice is accepted even when no trained template matches it (previously rejected with INVALID_DOCUMENT_TYPE_ADVICE), as long as the generic reader is enabled; advice for any other country, or naming several countries, is checked against the trained templates as before, and so is all advice when the generic reader is disabled. A template that matches the advised page, an MRZ, or a decodable PDF417 on it keep their routes - a decodable AAMVA barcode still classifies the page as the back with the state or province as edition.

Document sources and configuration. The document sources of the request decide which routes are open. The routes for cards without a template - barcode classification, front pairing and the advised front - need the GLOBAL_OCR source, requested directly or implied by VIZ; BARCODE alone classifies nothing and only adds the decoded barcode values to the response. The generic read of a template-classified page needs VIZ; with GLOBAL_OCR alone such a page keeps its template read. TEMPLATE_FIRST (see above) restores the template read and FULL_SUPPORT for cards with a template. With the generic reader disabled, templates are always used and US and Canadian cards without a template are not recognized.

Latency. The generic document reader is much slower than a template read. Measured on a development machine without the VNNI instruction set, a page took roughly 7-12 s to read, against about 3 s for a template read, and the confirming upload of a barcode-classified pair adds one such read. The reader is optimised for VNNI (AVX-512 VNNI, available e.g. on AWS c7a instances); on such hardware with a recent ONNX Runtime the latency is expected to drop, but this has not been measured yet. Size the request timeouts of your integration for the slower case, and prefer VNNI-capable nodes for DIS when US or Canadian documents are a significant share of your traffic.

Document Classification#

Before processing the OCR, the images of the document pages needs to be classified. The classification uses two methods. Visual classification is searching for an image template (among the supported document types) with highest similarity to the processed page. MRZ classification reads the country ISO code from the MRZ zone (if present) and identifies the number of rows and letters in it to identify the document type.

The document page image can be classified as one of these 3 types:

• Front page
• Back page
• Page of an unknown document

When a new page is uploaded, it gets a list of possible classification candidates. Based on these candidates, it is decided, whether it is a front or a back page (or unknown document). If there was an image with same document page type (front/back) uploaded before, it gets replaced. So only the last submitted pages of each type contribute to the classification of the ID document.

When deciding the classification of the document, all the classification candidates are tried to be matched for the same document type between the front page and the back page. A classification pair with the highest sum of confidences decides the classification of the whole document. After OCR there might be an extra check to validate, if the classification was correct based on the extracted texts. If a misclassification was detected, the classification pair with the second highest sum of confidences is chosen for the document classification.

Data normalization#

DIS provides a standardized set of text fields, so that documents of different types can be evaluated with the same logic. Text fields such as dates are normalized to ISO format.

To find out which text fields are returned for a specific document type, use the GET metadata API call.

Document authenticity#

DIS provides technology to check the authenticity of an identity document, more about it is in the Document Authenticity Evaluation.

Document photo quality check#

The document page photo quality check is provided if the corresponding API call is used. It checks if the photo of the document has the correct brightness, sharpness, doesn’t contain hotspots, has enough background borders around the document in the image, and is the correct size.

The document page photo quality check result contains detection confidence and the coordinates of the document in the image.

Image requirements#

• The supported image formats are JPEG and PNG
• The document image must be large enough - document card width should be approximately 750 pixels on the image
• The document card edges must be clearly visible and be placed at least 10 px inside the image area
• The image must be sharp enough for the human eye to recognize the text
• There should be no glare on the document card that obscures the text or photo
• The image should not contain objects or backgrounds with visible edges. This can confuse the detection process
• The document should not be a specimen

Example of a sufficient quality document

Example of a not suitable image - the borders around the document are cropped

Further examples of not suitable images due to low quality are in the technical documentation section document page quality.