Your Contracts Folder Is Not a Search Problem

Sitting on a folder of documents nobody can query?

I agree to the Privacy Policy Talk Extraction Pipelines

Two hundred supplier agreements. PDFs, one folder, a shared drive. Legal asks a question: which of these renew automatically in Q1, and which need notice served before December?

Everyone’s first instinct is a vector store. Embed the PDFs, retrieve the relevant chunks, let the model answer.

Try it on that exact question and watch it fail.

Retrieval returns what resembles your query. “Which contracts auto-renew in Q1” resembles nothing in particular — it asks about all two hundred at once, and it filters on a field. There is no top-k that means “all of them.” You get five chunks that happen to mention renewal, a confident paragraph about five contracts, and no signal at all that the other 195 were never opened.

Retrieval is what you reach for when you don’t have a schema. This folder has one. Every agreement in it has a counterparty, a start date, a term, a notice period. Nobody has written it down yet.

So write it down. That’s the whole build, and it takes two tools.

Two hundred agreements, every fact in them true and none of them addressable. Photo by Wesley Tingey on Unsplash.

Three kinds of question

Carve the questions before designing the storage. A contracts folder gets asked three different things, and they are not the same shape.

The middle row is the one that breaks retrieval, and it’s the row your finance team actually cares about. It needs every document, not the most similar ones. A WHERE clause, not a cosine distance.

The third row needs the opposite: not fields, but the sentences themselves, in a form you can read without opening Acrobat.

Two kinds of question, two kinds of artifact. So the pipeline produces both.

Think of it as what a paralegal did before any of this existed. They photocopied the contract so it could be read without pulling the original from the cabinet, and they wrote an index card so it could be found without being read. The photocopy is a markdown file. The index card is a row of JSON.

The drawer does not hold the book. It holds the one row that tells you which book to pull. Photo by Jan Antonin Kolar on Unsplash.

The two tools

The agent gets exactly two, and both of them write files and return a summary. Neither one hands the model a pile of contract text and hopes.

The first turns PDFs into markdown. No model involved — this is a parser.

from pathlib import Path
from pypdf import PdfReader
from langchain.tools import tool

CONTRACTS = Path("contracts")
MARKDOWN = Path("extracted")


@tool
def extract_contracts() -> str:
    """Convert every PDF in the contracts folder to markdown. Skips files that
    are already extracted. Returns how many were written."""
    MARKDOWN.mkdir(exist_ok=True)
    written = 0

    for pdf in sorted(CONTRACTS.glob("*.pdf")):
        out = MARKDOWN / f"{pdf.stem}.md"
        if out.exists():
            continue
        reader = PdfReader(str(pdf))
        text = "\n\n".join(page.extract_text() or "" for page in reader.pages)
        out.write_text(f"# {pdf.name}\n\n{text}")
        written += 1

    return f"Extracted {written} new contracts, {len(list(MARKDOWN.glob('*.md')))} total."

The second reads that markdown and produces the index cards. This is where the model works, and it works on one document at a time against a fixed schema.

import json
from pydantic import BaseModel, Field
from langchain.chat_models import init_chat_model

INVENTORY = Path("inventory.json")


class Contract(BaseModel):
    vendor: str = Field(description="The counterparty, not our own company")
    effective_date: str = Field(description="ISO date the agreement starts")
    term_months: int
    notice_days: int = Field(description="Days of notice required to terminate")
    auto_renew: bool
    governing_law: str
    annual_value_eur: int | None = Field(description="Null if not stated")


extractor = init_chat_model("anthropic:claude-opus-5").with_structured_output(Contract)


@tool
def build_inventory() -> str:
    """Read the extracted markdown and produce a JSON inventory, one record per
    contract. Returns the full inventory — answer questions from it."""
    rows = json.loads(INVENTORY.read_text()) if INVENTORY.exists() else []
    done = {row["file"] for row in rows}

    for md in sorted(MARKDOWN.glob("*.md")):
        if md.name in done:
            continue
        record = extractor.invoke(
            "Extract the contract fields from this agreement.\n\n" + md.read_text()
        )
        rows.append({"file": md.name, **record.model_dump()})

    INVENTORY.write_text(json.dumps(rows, indent=2))
    return json.dumps(rows)

Then the agent, which is almost nothing:

from langchain.agents import create_agent

agent = create_agent(
    model="anthropic:claude-opus-5",
    tools=[extract_contracts, build_inventory],
    system_prompt=(
        "You answer questions about a folder of business contracts.\n"
        "Run extract_contracts first if nothing has been extracted yet, then "
        "build_inventory to get the structured records. Answer from the "
        "inventory. Cite the file name for every contract you mention."
    ),
)

result = agent.invoke({"messages": [{
    "role": "user",
    "content": "Which contracts auto-renew and need under 60 days' notice?",
}]})

print(result["messages"][-1].content)

Three design choices in there are doing the real work.

Both tools are idempotent. Each skips what it has already done. You will run this pipeline more than once — a new contract lands, a field gets added — and re-running has to be cheap enough that nobody thinks twice about it.

The schema lives in one Pydantic class. That class is the product. Choosing notice_days: int over notice_period: str is what makes “under 60 days” answerable, and no amount of prompt engineering recovers it if you get that wrong.

Extraction runs per document, not per corpus. One contract, one call, one record. That keeps each extraction inside a small context, makes failures land on a single file instead of the batch, and means a schema change re-runs against markdown you already have.

What the inventory actually buys

I built 200 synthetic four-page contracts and measured it. Full script and provenance below.

inventory JSON   :    39,241 chars  (~9,810 tokens)
markdown corpus  : 2,652,727 chars  (~663,181 tokens)
ratio            : 67.6x

That’s the number the whole design answers to. Every aggregation question costs about 9,800 tokens instead of 663,000. The inventory fits in a prompt with room to spare; the corpus technically fits too, in a million-token window, and costs roughly seventy times as much to ask a question nobody needed the prose for.

Sixty-eight times is not a speed optimisation. It’s the difference between a question you ask casually and a question you file a ticket for.

Now the number that argues against me, which I’m printing because I went looking for it.

I assumed the markdown layer would justify itself on re-extraction cost — that when the schema changes, reading .md beats re-parsing 200 PDFs. It does, and the saving is nothing:

parse 200 PDFs -> text : 4.026s   (20.1ms per contract)
read 200 .md files     : 0.002s   (0.009ms per contract)
speedup                : 2148x
disk                   : 1.15 MB of PDF -> 2.65 MB of markdown
tokens saved           : 0

A 2,148× speedup that saves you four seconds, once, per schema revision — against an extraction pass that spends 663,000 input tokens and several minutes of model latency, and costs you 2.3× the disk. The markdown layer buys no meaningful time and not one token. On the efficiency argument, one-pass PDF-to-JSON wins outright.

Estimator and its bias: median wall-clock over five warm runs, single process, no concurrency. It is biased low against production — page cache is hot, the disk is local NVMe, and the documents are clean generated PDFs with a real text layer. Real contract folders sit on network storage and run longer. The direction of the bias makes my own case weaker, not stronger, which is why I’m reporting the median rather than the cold first run.

Arch Linux 7.0.10 · Python 3.14.6 · pypdf 6.16.1 · reportlab 5.0.0
i9-12900HK · n = 200 contracts, 4 pages each · seed 4444 · 2026-08-15

The JSON is what makes the question cheap. The markdown sitting next to it is what makes the answer checkable.

What it costs, and where it stops

Take the case against seriously, because it is good. One pass, PDF straight to JSON, is half the code, has no second artifact to keep in sync, and cannot go stale. My own benchmark says the intermediate file saves nothing measurable. An engineer who built it that way would be making a defensible call and could point at my numbers to defend it.

Here is what they’d be giving up, and it isn’t speed.

When the inventory says notice_days: 60 and the renewal is wrong, someone has to find out whether the model misread the contract or the contract genuinely says sixty days. With markdown on disk that’s a grep. Without it, it’s re-running an extraction and hoping it’s deterministic, or opening the PDF and reading — which is the job the pipeline was built to eliminate.

That matters because of who is holding the answer. The person querying this is in contracts or legal ops, and they sign off on a renewal date. An answer they cannot trace back to a sentence is an answer they have to take on faith, from a system whose reasoning is unavailable to them. They will not run the extraction again to check. They will either trust it, or quietly go back to opening the PDFs — and the second one is what actually happens.

Every extracted field is a claim. Someone eventually has to check one against the page it came from. Photo by Harshit Suryawanshi on Unsplash.

The markdown layer buys no speed and no money. It buys the ability to check. That is the entire argument, and it’s worth being honest that it’s an argument about accountability rather than performance.

Two boundaries on all of this.

My number is measured on the easy case. pypdf reads a text layer. Scanned contracts have none — those need OCR or a vision model, at seconds and real money per page instead of 20 milliseconds. On that input the arithmetic inverts completely: caching the extracted text stops being a rounding error and starts being the reason the second schema revision is affordable at all. The benchmark that undercuts my recommendation only undercuts it on the input where it matters least.

And the inventory is only as good as the division. Everything here rests on the schema being a decent carve of what people ask. Add a field nobody queries and you’ve paid for extraction you’ll never use. Miss one they need and you’re re-running the whole corpus. That decision is not a modelling problem and no framework will make it for you — it comes from watching what your team asks the folder, which is an afternoon of listening rather than an afternoon of coding.

Closing

The interesting thing about this build is how little of it is the agent.

Two tools, one Pydantic class, a create_agent call that is genuinely five lines. The framework contributes almost nothing you couldn’t write yourself in an afternoon, and that is the correct amount for it to contribute.

What made the questions answerable was deciding they were three different kinds of question, and that two of them wanted a schema rather than a search index. The rest followed.

Most teams reach for embeddings first, because “search the documents” is the phrase that comes to mind. Then they discover it can’t count.

The benchmark

Generates the corpus and measures both figures. Deterministic under the seed.

"""Two-stage contract extraction: what does the .md layer actually buy?
pip install pypdf reportlab
"""
import json, random, statistics, time
from pathlib import Path
from pypdf import PdfReader
from reportlab.lib.pagesizes import A4
from reportlab.lib.styles import getSampleStyleSheet
from reportlab.platypus import Paragraph, SimpleDocTemplate, Spacer

SEED, N_CONTRACTS, REPS = 4444, 200, 5
ROOT = Path(__file__).parent
PDF_DIR, MD_DIR = ROOT / "contracts_pdf", ROOT / "contracts_md"

VENDORS = ["Acme Logistics", "Northwind Traders", "Contoso Cloud", "Fabrikam Metals",
           "Litware Analytics", "Tailwind Freight", "Proseware Media", "Adventure Foods"]
LAWS = ["Romania", "England and Wales", "Delaware", "Germany", "Netherlands"]
CLAUSE = ("The Supplier shall provide the Services with reasonable skill and care and in "
          "accordance with the standards of the industry. The Customer shall pay all "
          "undisputed invoices within thirty (30) days of receipt. Neither party shall be "
          "liable for indirect or consequential loss howsoever arising, whether in contract, "
          "tort or otherwise, save in respect of death or personal injury caused by negligence. "
          "Each party shall indemnify the other against all claims arising from a breach of "
          "its confidentiality obligations under this Agreement. ")


def generate_pdfs():
    rng = random.Random(SEED)
    PDF_DIR.mkdir(parents=True, exist_ok=True)
    styles, facts = getSampleStyleSheet(), {}

    for i in range(N_CONTRACTS):
        name = f"contract-{i:04d}.pdf"
        fact = {
            "vendor": f"{rng.choice(VENDORS)} {rng.randint(1, 99)}",
            "effective_date": f"202{rng.randint(3,5)}-{rng.randint(1,12):02d}-{rng.randint(1,28):02d}",
            "term_months": rng.choice([12, 24, 36]),
            "notice_days": rng.choice([30, 60, 90]),
            "auto_renew": rng.choice([True, False]),
            "governing_law": rng.choice(LAWS),
            "annual_value_eur": rng.randrange(10_000, 500_000, 1_000),
        }
        facts[name] = fact

        flow = [
            Paragraph("MASTER SERVICES AGREEMENT", styles["Title"]),
            Paragraph(f"between DanStativa SRL and {fact['vendor']}", styles["Heading2"]),
            Spacer(1, 12),
            Paragraph(f"Effective Date: {fact['effective_date']}", styles["Normal"]),
            Paragraph(f"Initial Term: {fact['term_months']} months", styles["Normal"]),
            Paragraph(f"Notice Period: {fact['notice_days']} days", styles["Normal"]),
            Paragraph("Renewal: " + ("automatically renews" if fact["auto_renew"]
                                     else "expires at end of the Initial Term"), styles["Normal"]),
            Paragraph(f"Governing Law: the laws of {fact['governing_law']}", styles["Normal"]),
            Paragraph(f"Annual Value: EUR {fact['annual_value_eur']:,}", styles["Normal"]),
            Spacer(1, 12),
        ]
        for n in range(1, 13):
            flow.append(Paragraph(f"{n}. STANDARD TERMS", styles["Heading3"]))
            flow.append(Paragraph(CLAUSE * 2, styles["BodyText"]))

        SimpleDocTemplate(str(PDF_DIR / name), pagesize=A4).build(flow)

    (ROOT / "ground_truth.json").write_text(json.dumps(facts, indent=2))


def pdf_to_markdown(path):
    return "\n\n".join(p.extract_text() or "" for p in PdfReader(str(path)).pages)


def main():
    if len(list(PDF_DIR.glob("*.pdf"))) != N_CONTRACTS:
        generate_pdfs()

    MD_DIR.mkdir(parents=True, exist_ok=True)
    for pdf in sorted(PDF_DIR.glob("*.pdf")):
        (MD_DIR / f"{pdf.stem}.md").write_text(pdf_to_markdown(pdf))

    def time_parse():
        t = time.perf_counter()
        for pdf in sorted(PDF_DIR.glob("*.pdf")):
            pdf_to_markdown(pdf)
        return time.perf_counter() - t

    def time_read():
        t = time.perf_counter()
        for md in sorted(MD_DIR.glob("*.md")):
            md.read_text()
        return time.perf_counter() - t

    parse = statistics.median(time_parse() for _ in range(REPS))
    read = statistics.median(time_read() for _ in range(REPS))

    gt = json.loads((ROOT / "ground_truth.json").read_text())
    inventory = json.dumps([{"file": k, **v} for k, v in gt.items()], separators=(",", ":"))
    md_chars = sum(len(p.read_text()) for p in MD_DIR.glob("*.md"))

    print(f"parse PDFs -> text : {parse:.3f}s")
    print(f"read .md files     : {read:.3f}s")
    print(f"speedup            : {parse / read:.0f}x")
    print(f"inventory JSON     : {len(inventory):,} chars (~{len(inventory)//4:,} tokens)")
    print(f"markdown corpus    : {md_chars:,} chars (~{md_chars//4:,} tokens)")
    print(f"ratio              : {md_chars / len(inventory):.1f}x")


if __name__ == "__main__":
    main()

References

1. LangChain, “Agents”
2. LangChain, “Structured output”
3. LangChain, “Tools”

Sitting on a folder of documents nobody can query?

I agree to the Privacy Policy Talk Extraction Pipelines