Multimodal Invoice & Document Extractor
Turns invoices, PDFs, scans and photographs into schema-validated JSON using multimodal extraction — replacing template parsers that shatter the moment a layout shifts. Manual effort per document fell from about ten minutes to under one.
Engagement
Anonymized — delivered engagement
10 → <1 min
manual effort per document
The problem
What this replaced
The manual process, before
- 01Invoices, specification sheets and image-based reports arrive as unstructured files.
- 02Someone retypes them by hand.
- 03Or a regex/template parser handles them — until the vendor changes their layout.
Who has this problem
Operations teams, data entry teams, and backend engineers ingesting third-party assets.
Why it costs money
Manual extraction is slow, expensive and error-prone. Template parsers are cheap until they break, and they break silently, upstream of everything that depends on the data.
The build
What I built
Accepts PDF, PNG or JPG, uses Gemini's multimodal capability to read content and layout directly rather than OCR-then-regex, performs semantic schema parsing, and normalizes the result into strict, validated JSON.
Convert an arbitrary document image into machine-readable data with high fidelity, without a template per document type.
Capabilities
- Multimodal image and document reading
- Layout analysis
- Entity recognition
- JSON schema enforcement and type validation
How it runs
The workflow
Trigger
A raw file — PDF, PNG or JPG — with an optional target schema.
- 01Ingest and preprocess the document or image
- 02Run multimodal extraction over textual and spatial content
- 03Parse extracted content into predefined entity keys and fields
- 04Enforce schema structure and validate data types
Output
A clean, validated, structured JSON payload ready for a database or pipeline.
Architecture
How it fits together
Components
- File intake (PDF / PNG / JPG)
- Preprocessing and normalization
- Gemini multimodal extraction
- Schema parser and validator
- Structured JSON output
Data flow
01
File in → preprocess
02
Multimodal extraction — text and layout together
03
Map to entity keys → enforce schema
04
Validated JSON out
The guardrail
What it is not allowed to do
Schema validation is the contract.
Extraction is not trusted on the model's word. Output must satisfy a strict schema with typed fields — numbers, dates, strings — before anything downstream sees it. A model that invents a field fails validation rather than quietly corrupting a database.
Stack
What it is built on
- Model
- Gemini — multimodal
- Framework
- LangChain
- Validation
- Strict JSON schema enforcement
- Cloud
- AWS EC2
Hard parts
What made it difficult
- 01
Holding extraction fidelity across wildly inconsistent document layouts.
- 02
Preserving spatial relationships — a number means nothing without knowing which column it sat in.
- 03
Enforcing type validity on fields the source document formats inconsistently.
Outcome
What changed
10 → <1 min
manual effort per document
Manual effort per document fell from about ten minutes to under one.
Replaces manual typing and ongoing regex-script maintenance with semantic parsing.
End-to-end conversion from document or image file to database-ready JSON.
Delivered and running against real invoice and document intake.
Written as a de-identified case study. The client's identity and the contents of any processed document are not disclosed.
Limited availability
Tell me the problem.
I’ll tell you if an agent is the answer.
A 30-minute call, no pitch. Describe the manual loop that hurts and I’ll give you a straight read on whether this is worth building — including when the honest answer is a script, not an agent.
30 minutes · video call · no pitch