Multimodal Invoice & Document Extractor
Turns invoices, PDFs, scans and photographs into schema-validated JSON using multimodal extraction — replacing template parsers that shatter the moment a layout shifts. Manual effort per document fell from about ten minutes to under one.
- Trigger
- Agent
- Tool / data
- Guardrail
- Human
⊘ Schema validation is the contract.
The problem
What this replaced
- 01Invoices, specification sheets and image-based reports arrive as unstructured files.
- 02Someone retypes them by hand.
- 03Or a regex/template parser handles them — until the vendor changes their layout.
Who has this problem
Operations teams, data entry teams, and backend engineers ingesting third-party assets.
What it costs
Manual extraction is slow, expensive and error-prone. Template parsers are cheap until they break, and they break silently, upstream of everything that depends on the data.
The build
What I built
Accepts PDF, PNG or JPG, uses Gemini's multimodal capability to read content and layout directly rather than OCR-then-regex, performs semantic schema parsing, and normalizes the result into strict, validated JSON.
Convert an arbitrary document image into machine-readable data with high fidelity, without a template per document type.
Capabilities
- 01Multimodal image and document reading
- 02Layout analysis
- 03Entity recognition
- 04JSON schema enforcement and type validation
How it runs
The workflow
- in
Trigger
A raw file — PDF, PNG or JPG — with an optional target schema.
- 01
Ingest and preprocess the document or image
- 02
Run multimodal extraction over textual and spatial content
- 03
Parse extracted content into predefined entity keys and fields
- 04
Enforce schema structure and validate data types
- out
Output
A clean, validated, structured JSON payload ready for a database or pipeline.
Components
- File intake (PDF / PNG / JPG)
- Preprocessing and normalization
- Gemini multimodal extraction
- Schema parser and validator
- Structured JSON output
The guardrail
What it is not allowed to do
⊘ Boundary
Schema validation is the contract.
Extraction is not trusted on the model's word. Output must satisfy a strict schema with typed fields — numbers, dates, strings — before anything downstream sees it. A model that invents a field fails validation rather than quietly corrupting a database.
Stack
What it is built on
- Model
- Gemini — multimodal
- Framework
- LangChain
- Validation
- Strict JSON schema enforcement
- Cloud
- AWS EC2
Hard parts
What made it difficult
- 01
Holding extraction fidelity across wildly inconsistent document layouts.
- 02
Preserving spatial relationships — a number means nothing without knowing which column it sat in.
- 03
Enforcing type validity on fields the source document formats inconsistently.
Outcome
What changed
10 → <1 min
manual effort per document
Manual effort per document fell from about ten minutes to under one.
Replaces manual typing and ongoing regex-script maintenance with semantic parsing.
End-to-end conversion from document or image file to database-ready JSON.
Delivered and running against real invoice and document intake.
Written as a de-identified case study. The organisation's identity and the contents of any processed document are not disclosed.
Get in touch
Building an agent that has to survive production?
I write about MCP design, evals and guardrails on LinkedIn most days. If you're working on something similar, I'm happy to talk shop.