On-premise document verification & field extraction

The problem

A Tunisian bank processes securities-order and fund-subscription forms on paper. Before an order can be acted on, four marks have to be present and correct - the agency stamp, the bank signature, the words "signature conforme," and the client's own signature - and around 30 fields have to be read off the form and into the back office.

Correct here does not mean approximately right. A misread account number or amount is a compliance incident, not a rounding error, so the system had to be able to say when it was not sure rather than produce a confident answer either way.

The constraint

The service runs on-premise, inside the bank's own network. Client documents never leave the building, so there is no cloud OCR and no external API anywhere in the pipeline.

Scans arrive as they come: skewed, rotated, photocopied, and sometimes stamped directly over the field the stamp obscures. Because a wrong answer costs more than no answer, the system needed a defensible way to abstain rather than guess.

What I built

Two paths over the same document. A trained detector locates the four mandatory marks and returns boxes with confidences. Separately, a schema describes where each of the roughly 30 fields sits on each form variant, and zone OCR reads those regions, with pattern validation where a field has a known shape.

The two paths combine into a three-tier verdict and an explainable 0-100 trust score, so a reviewer can see which mark or which field pulled the score down rather than being handed a bare number. The service is wrapped in FastAPI with Pydantic models at the boundary, and a Streamlit console lets an operator page through results and inspect individual documents.

What it measured

96.1% field-extraction accuracy on the demo set, and 91.3% on a held-out 115-document test split. The gap between those two is the honest number: the demo set is easier, and both are published rather than only the flattering one.

Hardening ran as six numbered rounds. Each round had to show zero regressions against the previous round's committed scorecard before it could merge, so improving one field at the cost of another did not count as progress. The hybrid strategy is what moved the fields pattern-matching had never handled: the form date went from not being extracted at all to being read on more than half of documents, and nationality went from under half to over eighty percent, neither at the cost of a field that already worked.

The figures come from an evaluation harness that scores detector boxes, extraction, rejection behaviour and trust calibration, and writes a scorecard file into the repository on every run. A synthetic document generator supplies additional cases. Every number quoted here is read from a committed scorecard rather than from memory.

What I'd change

The five-point gap between the demo set and the held-out split is mostly form variants the schema had never seen. I would invest in variant discovery earlier instead of hardening further against the variants I already had.

I would also calibrate the trust score against reviewer decisions rather than against extraction correctness. As built it answers "how sure is the model," when what an operator actually needs to know is "how likely is a human to overturn this."

At a glance

• 96.1% extraction accuracy on the demo set and 91.3% on a held-out 115-document test split, reached over six hardening rounds - each one required to show zero regressions against the previous scorecard before it could merge.
• Combined a trained detector for the mandatory marks with schema-driven zone OCR for the form fields. The hybrid recovered fields that pattern-matching alone had never handled at all, without losing accuracy on any field that already worked.
• Built the evaluation harness that produces those scorecards, plus a synthetic document generator - so every number published traces back to a committed file.

Stack