All work

Case study ·

Financial document intelligence

A document AI pipeline that turns mortgage files, from W-2s and paystubs to 136-page tax returns, into structured data where every value is traced back to its page and position.

  • 680msp95 extraction, down from 3.2s
  • −45%LLM token spend
  • 99.95%platform uptime
  • 5document types
  • Python
  • Gemini
  • pgvector
  • PostgreSQL
  • Redis
  • Cloud Run
  • Terraform
  • Bazel
  • Langfuse

Context

Mortgage underwriting depends on a borrower's documents: W-2s, paystubs, bank statements, driver's licenses and federal tax returns. Reviewers used to read each one and retype the values they needed, and a single file can run to a 50-page bank statement or a 136-page tax return.

Problem

Every step after a document arrived was manual. Reviewers validated each document, matched it against the rest of the file and filled the loan forms by hand, which made underwriting slow and its throughput tied to headcount.

Role

I designed and built the document pipeline and managed the platform's cloud infrastructure in Terraform.

Architecture

  1. Intake. A task on a Redis queue points to a PDF in cloud storage. Workers run on Cloud Run and scale with load.
  2. Rendering. Pages render in a process pool, which sidesteps Python's GIL. Images stay in memory across every stage; nothing touches disk.
  3. OCR. Pages are sent for OCR in batches, returning text and the bounding box of every word.
  4. Classification. Each page is matched against reference embeddings in pgvector (HNSW index). A BM25 keyword scorer runs alongside it, and the two scores are fused into one ranking.
  5. Grouping. Pages are grouped into documents by type, so a 40-page bank statement is extracted as one document.
  6. Extraction. Fixed IRS forms go to a deterministic rules engine first. Everything else, and anything the engine is unsure about, goes to Gemini in batches.
  7. Result. Every value is returned with the page it came from and its position on that page, so a reviewer can check it at a glance.

Pipeline versions are registered side by side, and each task names the version it wants. Shipping a new version never edits a running one.

Document intelligence pipeline architectureA task from a Redis queue points to a PDF in cloud storage. Pages are rendered in a process pool and OCR'd in batches, then classified by both vector similarity and BM25 keyword scoring, fused into one ranking. Pages are grouped into documents, extracted either by a deterministic rules engine or by batched, parallel Gemini calls, and returned as structured fields with page and position.Task queueRedis · PDF in storagePage renderingprocess poolOCRbatched callsVector matchpgvector · HNSWKeyword matchBM25 vocabularyScore fusionre-rank shortlistGroup pages by documentW-2 · paystub · 1040 · …Rules engineW-2 · 1040Gemini extractionbatched · parallelStructured resultvalue · page · position
The rules engine handles fixed IRS forms it is confident about; everything else falls back to Gemini.

Key decisions

Keyword scoring re-ranks the vector shortlist instead of voting. Financial documents share most of their vocabulary: amounts, dates, names. A hand-curated list of distinguishing phrases per document type, such as "adjusted gross" for a 1040 or "ending balance" for a bank statement, gives a sharper signal than raw text. BM25 can only re-rank categories the vector search already surfaced, a per-type minimum score keeps stray keywords out, and it runs in memory in a few milliseconds, so it adds no latency.

Batches with continuation context instead of one huge prompt. A long document cannot fit in one call. The first batch extracts every field it sees. Later batches run in parallel and receive a compact summary of what has already been captured, with coordinates stripped, and are told to return nothing for fields already found. Type-specific hooks handle the hard merges, such as accounts spread across a bank statement.

Rules before the model for fixed forms, with precision first. A W-2's box labels are fixed by law even though their positions vary by employer. The rules engine finds each printed label, reads the value in a zone sized relative to that label, validates its type and checks the Social Security and Medicare arithmetic. It answers only when every check passes; otherwise the document goes to Gemini exactly as before. Rollout moves from model-only, to running both and comparing, to rules-first.

Results

  • p95 extraction latency down from 3.2s to 680ms.
  • LLM token spend down 45%.
  • 99.95% platform uptime across separate development, staging and production environments.
  • Validation, cross-document matching and form filling now run automatically; reviewers report far less manual work.
  • Every extracted value carries its page and position, so review becomes checking rather than retyping.

What's next

Planned: agents that walk a pre-approved loan through to closing. They will request new documents when a borrower's scenario changes, highlight inconsistencies across the file, and act on the platform through APIs exposed as Model Context Protocol tools.

What I'd change

Most documents still go to a hosted LLM, which sets the floor on latency and cost and means borrower data leaves our infrastructure for every extraction. I would train our own extraction model on the documents the pipeline has already processed, where every value is labelled with its page and position. A smaller in-house model would be faster and cheaper per page, and sensitive financial data would never leave our environment.