Guru Balamurugan AI Engineer
Menu
Back to projects

In production Document extraction

Production-grade financial document extraction

DocExtract turns messy private-equity PDFs into structured, auditable data. It combines layout-aware parsing, LLM classification, decomposed extraction, deterministic validation, RAG, observability, and human-in-the-loop review so financial operations teams can move from manual document review to governed automation.

Built for capital calls, distribution notices, and other private-market documents with inconsistent templates, sparse public data, and high accuracy requirements.

View architecture diagram
Role
Sole AI engineer: design, build, evaluation
Domain
Private-market fund operations
Core stack
FastAPI · LangGraph · AWS Bedrock · Docling · Pydantic · Postgres/pgvector · Phoenix

Impact

  1. ~1 month → <1 week

    Manual review cycle

  2. 50%+ smaller

    Review team

  3. 91%

    Evaluation accuracy vs test set

  4. +1.4x cost

    For stronger supervisor-worker extraction

  5. $2-3M + >$1M

    Vendor recs and internal build displaced

Metrics reflect internal evaluation and projected operational impact from the implemented workflow.

01

The problem

Private-market documents are not clean forms. They are visual, template-heavy PDFs where the values that matter are buried in tables, footnotes, legal language, and client-specific layouts.

A normal PDF → LLM → JSON pipeline breaks down because it cannot reliably answer five questions:

  • Where did this number come from?
  • Was the math checked?
  • Which model and prompt produced this result?
  • Should this output be auto-approved or sent to review?
  • Can a human correction improve the next run?

DocExtract was designed around those failure modes.

02

System architecture

Upload to extraction to governed storage. Classification and a RAG layer feed a supervisor-worker orchestrator; every output runs the validation and confidence gate before it can be committed.

DocExtract pipeline: PDF upload to S3, Docling preprocessing into layout-aware Markdown, classification with Haiku/SLM and a hierarchical-chunk RAG layer feeding a LangGraph supervisor-worker orchestrator with three workers (notice metadata; capital activity and amounts; dates and payment instructions), structured JSON output validated by Pydantic and Bedrock schemas, a deterministic validation stage, a confidence and quality gate that either approves and commits to Postgres/pgvector or routes to re-extraction and HITL review with a correction ledger, plus Phoenix and structlog observability.
View full size →

03

How it works

  1. 01

    Layout-aware ingestion

    Documents are uploaded through the API and stored in S3. Docling and Python preprocessing convert PDFs into structured Markdown while preserving tables, spatial relationships, and layout cues. Capital-call notices are not just text: the meaning of a number often depends on where it sits on the page.

  2. 02

    Classification before extraction

    The system first labels each document by type and template/client. A smaller model such as Haiku handles lower-cost classification, so expensive extraction models are reserved for where they add value. Those labels also seed future fine-tuning, template-specific evaluation, and cheaper operations over time.

  3. 03

    Decomposed extraction

    Instead of one large prompt extracting everything at once, a supervisor-worker pattern in LangGraph splits the task into three focused groups: notice metadata; capital activity and financial amounts; dates, payment instructions, and operational fields. This raised accuracy while only increasing cost by about 1.4x, a better spot on the cost-accuracy frontier.

  4. 04

    Symbolic-neural validation

    LLMs handle messy interpretation; code handles deterministic truth. After extraction, structured outputs pass through Pydantic schemas, Bedrock JSON schema enforcement, coded business rules, math checks, and reconciliation logic. The core design principle: the model extracts, the system verifies.

  5. 05

    Provenance by default

    Every extracted value is stored with provenance markers, such as bounding-box coordinates or exact text spans, so a reviewer can trace any number back to its place in the original document. That makes the output auditable rather than merely plausible.

  6. 06

    HITL review and correction logging

    Low-confidence or failed validations route to human review. Corrections are stored immutably alongside the original prediction, the corrected value, and a reason code, creating a feedback loop for evaluation, prompt improvement, template tuning, and future fine-tuning.

The model extracts. The system verifies.

04

What makes this production-grade

  1. Governed outputs

    Typed JSON schemas, Pydantic validation, Bedrock schema enforcement, and required provenance fields cut hallucination risk and keep every output reviewable.

  2. Deterministic checks

    Business rules, math validation, and reconciliation logic verify the fields the model should not be trusted to reason through on its own.

  3. Cost discipline

    Smaller models handle classification where possible; larger models are reserved for high-value extraction. The supervisor-worker design improved accuracy at only a 1.4x cost increase.

  4. Observability

    Phoenix, structured logs, model metadata, prompt versions, latency, and extraction events make failures diagnosable instead of invisible.

  5. Human-in-the-loop review

    The system does not assume full automation on day one. Low-confidence extractions route to review, and corrections become training and evaluation assets.

  6. Reproducibility

    SQLite checkpointing, immutable extraction events, prompt configs, model versions, and file hashes let the pipeline resume, replay, and audit past runs.

05

Data model

DocExtract stores more than the final JSON. It keeps the evidence trail behind each output, so the system can answer not only "what did we extract?" but also "why should we trust it?"

  1. 01 Original file hash
  2. 02 Parsed chunks
  3. 03 Model configuration
  4. 04 Extraction event
  5. 05 Structured result
  6. 06 Provenance markers
  7. 07 Human corrections

06

Retrieval design

DocExtract uses context-enriched hierarchical chunking to improve retrieval over financial PDFs. Each chunk carries global document context, such as fund name, GP, notice date, and document type, and localized context describing the role of that section.

The retrieval layer combines four signals:

  • Dense vector search for semantic matches
  • Sparse keyword search for exact identifiers, amounts, and fund names
  • Reciprocal rank fusion for candidate selection
  • Reranking for final precision before extraction

This avoids relying on vector search alone, which is often weak for exact numbers, proprietary acronyms, and alphanumeric identifiers.

07

Technical stack

Backend / orchestration
FastAPI, Docker, LangGraph, SQLite checkpointer
Document storage
AWS S3
Parsing
Docling, Python preprocessing, layout-aware Markdown
Models
AWS Bedrock, Claude Sonnet, Claude Haiku / SLM classification
Validation
Pydantic, Bedrock JSON schemas, deterministic business rules
Retrieval / storage
Postgres, pgvector, JSONB, hierarchical chunks
LLMOps
Phoenix, structlog, extraction event telemetry
Migrations
Alembic
Review loop
HITL correction ledger, reason codes, confidence gates

08

What I learned

The biggest lever was refusing to trust a single prompt. Splitting extraction into a supervisor and three focused workers cost about 1.4x more per document but moved accuracy to 91%, which is the tradeoff a financial-operations team will take every time.

Deterministic validation turned out to be load-bearing, not a nicety. Schema enforcement, math checks, and reconciliation are what let the output be approved without a human reading every line, and provenance is what makes the approval defensible later. The model is the interpreter; the system is the auditor.