Kadam Parikh

Projects / Built · Case study

Comparing five insurers’ policies, clause by clause

A proof-of-concept that reads five insurers’ motor policies, written in Montenegrin, and says for about 45 named risks whether each is covered as standard, only as a paid add-on, or excluded.

Context
Regional insurance brokerage
My role
Sole engineer, two-week R&D sprint
Status
Proof of concept, not in production
Focus
Messy documents to checked answers

The problem

A broker advising a client on comprehensive motor cover has to cross-reference five insurers’ policy documents, each worded differently, to answer questions like “does this insurer cover theft of the car radio?” or “is speeding by more than 50 km/h an automatic exclusion?” Across about 45 risks and five insurers, that manual work is slow and easy to get wrong. The question was whether an LLM could do it reliably enough to trust.

The data

  • Eight policy PDFs, 69 pages in all, from five insurers: Generali (20 pages), Grawe (11), Sava (12 plus a 4-page breakage supplement), Lovćen (8 plus a 1-page supplement) and Uniqa (9 plus a 4-page supplement).
  • Every document is in Montenegrin, set in two columns and full of diacritics. Each insurer words and numbers its articles differently, so the same risk sits in a different place, under different terms, in each one.
  • Coverage is often split across the general terms and a separate supplement. An answer for one insurer can depend on reading both.
  • The answer sheet is an expert’s classification of about 45 risks for each of the five insurers: 225 decisions to score against.

What I did

This was a solo project. Everything below is my work.

  • Took accuracy from 68% to 91% against an expert’s answers by measuring each approach’s ceiling before moving to the next, rather than tuning one design indefinitely.
  • Rejected two off-the-shelf options before writing a custom pipeline: a managed file-upload RAG setup, and Azure’s document-extraction service, which did not hold up on two-column, diacritic-heavy PDFs.
  • Designed the final pipeline as five stages, with a strict split between one stage that only describes what a clause says and a separate stage that alone decides the coverage status.
  • Replaced a three-vote model panel that was structurally rigged with a two-model panel across different model families, delivered through schema-constrained tool calls instead of free-text JSON.
  • Made every stage resumable and dry-runnable with the model calls stubbed out, so a multi-hour, multi-dollar run never repeats finished work and structural changes can be checked before spending API budget.
  • Built a bilingual (Montenegrin and English) intake form prototype for brokers, sharing one risk taxonomy with the pipeline so the two could not drift apart.

The final pipeline

Pipeline: chunk documents, extract evidence per chunk, bucket results with plain code, arbitrate conflicts with two model families, verify, with scoring against an expert-curated answer sheet underneath.Chunk3 chunk modesExtractevidenceBucketplain codeArbitratetwo familiesVerifysecond checkScored against an expert-curated answer sheetevery stage cached and resumable · dry-run with model calls stubbed
Simplified view. Stage three is ordinary code: it sorts extracted evidence into gaps, agreements and conflicts before any model is asked to decide.
  1. Chunk. Splits each policy into clause-sized pieces, in one of three chunking modes.
  2. Extract. For each chunk, the model records what the clause says about each risk, without deciding coverage.
  3. Bucket. Plain code sorts the extracted evidence into gaps, agreements and conflicts.
  4. Arbitrate. Two models from different families decide the coverage status where evidence conflicts.
  5. Verify. A second check challenges the decided status before it is accepted.

Key decisions

  • Stop retrieving. Semantic vector search topped out at about 68%, and a hand-built full-text index with diacritic folding at 82%, both limited by missing the right clause. Skipping retrieval and pasting whole documents in reached 86–88%; mapping every chunk once and arbitrating reached 91%. For documents small enough to fit in context, retrieval was pure downside.
  • Real second opinions, not more votes. The earlier three-vote design asked one model twice, so ties always favored that model agreeing with itself. Two calls to genuinely different model families replaced it.
  • One targeted follow-up beat more redundancy. Merging three independent extraction passes lowered accuracy (78.5% to 73.8%), because it merged their mistakes as readily as their findings. A single “did you miss anything?” follow-up raised it (76.9% to 86.2%).
  • Batches of 15 risks, not all 45. A direct run comparison showed smaller risk lists per call extracted more reliably, at three times the call volume.
  • A rule in code where prompting stopped working. One “unless otherwise agreed” clause pattern was mistagged in 13 of 14 occurrences and stayed unstable across repeated samples. It is now matched by a regex and tagged before the model’s own answer is consulted.
  • Label, then map. The model emits a semantic label such as “standard coverage”, never a business code. A fixed dictionary maps label to code afterwards, so changing terminology never touches a reasoning prompt.

Outcomes

68% → 91%Accuracy against the expert’s answers, from off-the-shelf RAG to the final pipeline
76.9% → 86.2%From one targeted self-review follow-up, verified in a controlled comparison
15 of 225Decisions that flipped between identical runs with no code change

How success was measured

  • Ground truth was an insurance expert’s own answers, per insurer and risk. Every approach was scored against that sheet, never against another model’s output.
  • Because identical runs flipped 15 of 225 decisions, a single before-and-after comparison could fake a win or a regression. The rule became: run the baseline three times before trusting any change.
  • Fixes proven on a small test set were rechecked on the full 45-risk by 5-insurer matrix. That caught a prompt tweak that looked like a clean win and regressed the full set.