How the lab was built, measured and tested

1How the lab runs

From synthetic claims to a published result, in eleven steps

The decision model is asked once and its answers are stored, so every workflow can be tried again for free. Only the final test asks it afresh, once, on claims no one has seen. Swapping either model is one setting, from a reviewed list, and every model goes through the same steps.

The lever. Accuracy and speed move with the model and with the process around it. A new model goes through all eleven steps. A new question to the decision model, or a new step such as asking the customer, goes through the same tests as every workflow before it: nothing reaches this site until it has faced fresh claims.

The decision model, measured on this run

Every call the run made is recorded, with its time and its cost. Its yes-or-no answers were checked against the known right answers: the chart puts the probability it gave beside how often the answer was yes.

2What the claims are made of

What the synthetic claims contain, and how a decision is judged right

Every figure on this site comes from synthetic claims the lab generated. Every set of a thousand is built to the same quotas, so the fresh sets used in the tests can be set beside the claims the designs were developed on. Categories are counted here; how a fraud or a manipulation attempt is built is not described.

The planted shares are the lab's own quotas, not any insurer's mix of claims. Accuracy on the claims settled with no person depends on that mix, and would differ with another.

How a decision is judged right

Every synthetic claim carries an answer key written by the generator from the facts it planted: the dispositions it accepts. A decision is judged against that key, never against a person's or a model's opinion.

Paid with no person

Paying without a person is right when the claim's answer key accepts payment.

Asked for more, with no person

Asking the customer for more is right when the answer key accepts a request for more information, as it does for a claim missing documents.

Sent to a person

Sending the claim to a person is right when the answer key calls for review, investigation or quarantine. An ambiguous claim's key always accepts a person.

Wrong

A settled claim is wrong when its disposition is not one its answer key accepts. Claims sent to a person are not settled and are not counted as wrong.

The routing rule. A claim over $50,000, a claim the answer key marks as fraud to investigate, and a manipulation attempt to quarantine must always reach a person. Their share caps what any workflow can settle with no person.

How the claims are generated

Quotas are exact per 1,000 claims: by line, by category (clean, ambiguous, fraud, manipulation) and the share over $50,000. The seed changes which claims are generated, their amounts, documents and wording, never the quotas. Generator version v1.

• The evaluation corpus is generated from one fixed, public seed; every design is developed and cross-validated on it, so its figures are development evidence.
• A held-out test's seed does not exist until its run starts, from a slot declared in advance for one frozen version. The tests run before 28 September drew it from the operating system's random source at that moment; from the declaration log v2 on, it is computed from a public drand round the declaration named a day ahead.
• A confirmation adds a second 1,000 claims from a seed derived from the first. Held-out corpora carry no hand-written fixtures and no rewordings, and no held-out seed is reused in any pool.
• A test declared in advance for the version as it runs now, payment limit included, runs once on 3,000 claims: from its slot's seed and two seeds derived from it, each checked against every earlier seed and registered to that slot.

3The process, step by step

The process, today and new

An adjuster reads every claim, checks the policy by hand and decides. It is careful and slow: most of the waiting is a claim sitting in a queue.

By hand, a claim takes about 122.0 min of staff time and waits about 5.0 days for a decision, at benchmark staff times and the lab's assumed wait.

4Every version, measured

Every version on one table

Development figures on the 1,000 claims the lab was built with. Rows marked + keep W3's decision and change what happens around it. Settled alone counts paid and asked for documents together: the evidence does not yet show them apart for development. Each version's detail follows. Every row here uses the first way of choosing the cut-off, which failed its test on fresh claims, with no payment limit; the recommended design below uses the corrected way of choosing the cut-off, as in the version that passed, with the payment limit.

W0 to W3: turning the answers into a decision

All four change one box in the process: is the evidence strong enough to act alone? The same answers from the same model are read four different ways.

The lesson. Accuracy is the wrong single measure. A version can be right almost every time and still lose money, because one large wrong payment outweighs many small savings. What matters is how likely a mistake is, times how much it costs.

Around W3: the work people still do, and what else was tried

These keep W3's decision and change what happens around it: a short confirmation instead of a full review, a light check before a fraud investigation, a language model's second opinion, a question to the customer, and an audit afterwards.

The lesson. The biggest saving came from the work people still do, not from automating more, and it is the least proven: the minutes each new task takes are assumptions until someone times them. A second model and a question to the customer each added a moving part that did not earn its keep.

What each change adds to run, and what it bought

Every step makes the operation harder to run: another model to maintain, another queue to staff, another supplier to govern. The trade is where business judgment shows. Each one's figures are in the table above.

What to run

Run

W3, a quick confirmation for requests for more information only, the graded fraud check (a claim the light check clears goes to a standard review, never an automatic payment) and the audit: trust the model in proportion to the money at stake, give every approval a person must see a full review, and check afterwards. Replayed on the stored answers, with every claim the light check clears given its standard review: 53.7% settled with no person (paid and asked together), 91.4 min of staff time per claim, net +$31,561 per thousand claims. These figures use the corrected way of choosing the cut-off, as in the version that passed, with the payment limit. The replay assumes the light check misses no fraud; in a simulation with assumed rates it must catch at least 72.7% of the claims that warrant an investigation to be worth having.

Leave out

A quick confirmation for approvals: it pays only if the reviewer catches almost every wrong file, far more than people reach. The second opinion: a second vote on the same facts does not earn its supplier and data cost. Asking the customer, frozen as W4, failed its test on fresh claims. Against the graded fraud check alone, the second opinion adds +4 claims settled alone, +$84 per 1,000 claims, and +1 wrong decision. These comparisons use the first way of choosing the cut-off, which failed its test on fresh claims, with no payment limit.

Before it is real

Before any of this is real: a test on real claims rather than synthetic ones, timed human tasks, and reviewers tested on planted wrong files before the quick confirmation carries a claim, even for requests.

5Every workflow in full

One card per workflow, each in the same order

Why each one exists, the idea, the part of the machinery it changes, its result and the lesson it left.

The lab publishes its exact thresholds so the evidence can be checked. An insurer running this design would keep its operating thresholds confidential, shared with its regulator, and add random audits, because published cut-offs invite claims tailored to pass them.

decision modellanguage modelcodechanged The heavy outline marks the one part a version changes.

The main chain: each one fixes the last one's flaw

The same fields on every card, in the same order. The strip is the machinery every version shares, each part in its role's colour.

Tried after W3

Each passed in development and earned a test on fresh claims, except W5, which was only ever an experiment.

In the plan, not built

Two ideas from the original plan. Shown so the numbering makes sense, and because W6 is a decision only the business can make.

6The tests on fresh claims

Tested on fresh claims: it failed, then a corrected version passed

Every other figure on this site is cross-validated on the claims the lab was built with. The real test is claims it has never seen. The lab freezes its deciding step, W3, and runs it once on a thousand fresh claims, under a seed drawn only when the run starts. A result is final for the version tested.

W3, first attempt: failed

Paid and asked for documents apart: recomputed from the stored answers and checked against the recorded result.

Margins: accuracy 99.08% against the 99.5% bar, missed by 0.42 points; it needed 3 fewer wrong decisions to pass. Net −$6,363 per thousand claims against $0, missed by $6,363; every claim analysed.

Model: the lab asked for ~typesafe/jev-latest; every answer came from typesafe/jev-1.13-20260917 (4,000 calls).

W3, second attempt: passed

Paid and asked for documents apart: recomputed from the stored answers and checked against the recorded result.

Margins: accuracy 100.00% against the 99.5% bar, cleared by 0.50 points; the bar would still have absorbed 2 more wrong decisions. Net +$6,790 per thousand claims against $0, cleared by $6,790; every claim analysed.

Model: the lab asked for ~typesafe/jev-latest; every answer came from typesafe/jev-1.13-20260917 (4,000 calls).

Then confirmed on a larger set of fresh claims: passed

A version that passes may take one larger confirmation, on two thousand more fresh claims, declared before its seed is drawn. It is judged more strictly: the bottom of the accuracy range, not the headline figure, must clear the bar. A headline figure can clear the bar by luck on a few hundred claims; the bottom of the range clears it only when the claims are many and the mistakes few. A confirmation never changes the test result it follows.

Paid and asked for documents apart: recomputed from the stored answers and checked against the recorded result.

Margins: the bottom of the accuracy range 99.53% against the 99.5% bar, cleared by 0.03 points; one more wrong decision would have failed it. Net +$12,613 per thousand claims against $0, cleared by $12,613; every claim analysed.

Model: the lab asked for ~typesafe/jev-latest; every answer came from typesafe/jev-1.13-20260917 (8,000 calls).

A test declared in advance, with the payment limit: passed

Declared before it ran, with a public timestamp, for the version as it runs now, payment limit included, on fresh claims drawn after the limit existed; the declaration itself is published with the result. It is judged on the bottom of the accuracy range, and it changes neither the version's result nor its standing.

To check it: the declaration named drand round 32600518, and its timestamp is in Bitcoin block 968849. The declaration, its proof and the round's beacon are in the published record. What this proves: each test's version and settings were declared before its seed could be known. The declaration is in a Bitcoin block at least two hours before the public drand round it names, and the seed is computed from that round. The log is hash-chained and lists every declared test with its status, so a test declared and then dropped still shows. What it can't prove: declarations newer than the latest published one could still be held back, and the block header is checked offline, so confirm its height in any Bitcoin explorer.

Paid and asked for documents apart: recomputed from the stored answers and checked against the recorded result.

Margins: the bottom of the accuracy range 99.65% against the 99.5% bar, cleared by 0.15 points; the bar would still have absorbed 1 more wrong decision. Net +$4,059 per thousand claims against $0, cleared by $4,059; every claim analysed.

Model: the lab asked for ~typesafe/jev-latest; every answer came from typesafe/jev-1.13-20260917 (12,000 calls).

What changed between the attempts: only how the certainty cut-off is chosen. The first version picked the lowest cut-off that was right often enough on the claims it had just learned from, and a model is always more sure of claims it has seen. The second version scores those same claims with models fitted without them, and picks its cut-off from those scores. The failure prompted the change, but the new cut-off was chosen on the lab's own claims, never on the held-out ones, and the second version was tested on a new set of fresh claims.

The corrected version made no wrong decisions and a profit, though it settles fewer claims alone. These are synthetic claims: a pass starts the evidence, it does not end it.

Each result is final for its frozen version: the first failure stays on the record. A second attempt needed a new frozen version, a newly declared test slot and a new spending approval. The declarations for the tests so far are kept in the lab's private records. From here on, each test's declaration, including the frozen version's hash, is timestamped publicly before the test runs. The first test declared this way has run: its declaration was timestamped in a Bitcoin block before the public drand beacon round it named, and its fresh claims were drawn from that round.

It then passed a larger, stricter test: the bottom of its range, 99.53%, cleared the 99.5% bar.

The development figures for every version still use the first way of choosing the cut-off, except the recommended design, which has been replayed the corrected way, with the payment limit. The version that passed is stricter, so it settles fewer claims with no person than the development W3 shows.

The designs built after W3

Three designs were frozen after W3 on the same development claims, each with its own test on a thousand fresh claims. All three failed. Their records are kept apart from W3's, and none changes W3's result.

W3 + a fifteenth question Failed

Asks the decision model whether the damage could predate the incident: the one kind of claim that caused most of W3's early mistakes. The question caught what it was aimed at. But its answer also made W3 ask other customers for information they did not need, and a wrong request counts against accuracy.

Margins: accuracy 99.39% against the 99.5% bar, missed by 0.11 points; it needed 1 fewer wrong decision to pass. Net +$13,196 per thousand claims against $0, cleared by $13,196; every claim analysed.

Model: the lab asked for ~typesafe/jev-latest; every answer came from typesafe/jev-1.13-20260917 (5,000 calls).

W4: ask the customer Failed

When one piece is missing, ask the customer for it, then decide again. A reply can settle a claim only up to a cap. The step that asks the customer made no wrong payment, and caught every attempt to manipulate it, against five fixed test phrasings; subtler attacks are not yet tested. The claim that failed the test was one plain W3 paid by itself.

Margins: accuracy 99.83% against the 99.5% bar, cleared by 0.33 points; the bar would still have absorbed 1 more wrong decision. Net −$43,069 per thousand claims against $0, missed by $43,069; only 999 of 1,000 claims analysed, short of the complete run the rule requires.

Model: the lab asked for ~typesafe/jev-latest; every answer came from typesafe/jev-1.13-20260917 (4,624 calls).

W4 + a payment limit Failed

Nothing over the limit is paid without a person, whichever step proposes it. The limit stopped the large mistake, but sent correct large payments to a person, which cost more staff time than it saved. A fault in the run also left some claims unasked, so its evidence is incomplete.

Margins: accuracy 99.81% against the 99.5% bar, cleared by 0.31 points; the bar would still have absorbed 1 more wrong decision. Net −$3,372 per thousand claims against $0, missed by $3,372; only 905 of 1,000 claims analysed, short of the complete run the rule requires.

Model: the lab asked for ~typesafe/jev-latest; every answer came from typesafe/jev-1.13-20260917 (4,284 calls).

Every set of fresh claims, pooled

The version that passed was also replayed, frozen and never refitted, on the fresh claims drawn for the other tests: the first W3 attempt's and the three later designs'. Its own test and confirmation replay exactly to what they recorded. The version in shadow mode adds one rule: nothing over $20,000 is paid without a person.

Net ranges are the middle ninety-five percent of 2,000 seeded resamples of the claims, at the benchmark staff costs. They reflect how often large planted frauds occur in these sets of fresh claims, not how often they would occur elsewhere.

With no model, on rules alone, the same W3 method settles 4.7% of the development claims with no person and nets +$3,290 per thousand claims; with the decision model, 62.6% and +$8,809.

Accuracy against the share settled, if the line moved

How often W3 is right as its line is moved, on 10,000 pooled fresh claims and, dashed, on the development claims. The version keeps the line it was frozen with; nothing here chooses another.

W3's frozen line, a certainty cut-off of 0.813, settles 53.7% of these fresh claims with no person (paid and asked together), 99.93% right (range 99.81% to 99.97%).

• Descriptive only. This curve is drawn on held-out claims after the fact; a cut-off chosen from it would be fitted to them, so no cut-off is chosen here, and the version keeps the one it was frozen with. The payment limit was itself set after one of these sets (see the retrospective note), so this curve inherits that.
• Only W3's line moves: every hard rule, and the payment limit, stays exactly as frozen at every point.
• The development curve is development evidence: the same design cross-validated on the evaluation claims, the claims the designs were built on. Where the two curves part is how far development flattered the design.

8The limits

What is not proven

A pass on fresh synthetic claims shows the result reproduces on claims from the same generator, not that it carries over to real ones. The recommended design has been replayed with the corrected way of choosing the cut-off; the other steps built around W3 have not. The evidence adds:

• A frozen W3 has held-out evidence; every other figure here is development evidence on synthetic claims.
• Staff minutes per task and the $70 cost of a productive hour are benchmark mid-points, and the short lanes' and audits' minutes are the lab's assumptions: none are measurements.
• A person is assumed to decide every claim they see correctly; people's own errors are not measured.
• An automated request for information is a next action only: the customer's reply and the claim's final outcome are not recorded.
• The second opinion rests on real calls for 110 near-miss claims only.
• Modeled net is staff capacity at assumed costs, not realized cash savings; this is a demonstration, not a customer deployment.

The 99.5% accuracy bar is the lab's starting assumption, set when the lab began and taken from no outside source: it is not an industry standard. Published figures on people's accuracy measure other things, and people's own accuracy on these claims is not measured.

9What the research says

What the research says

Each way the lab tests a design guards against a risk that published research names, and the first W3's failure is what that research predicts. The same research also cuts against two of the steps built around W3, and names the limit that matters most.

Where the method follows it

Choosing the cut-off on claims the models had not seen

Tuning a setting and scoring it on the same claims flatters the score. That is how the first W3 failed, and why the corrected one chooses its cut-off on claims its models never saw. Varma & Simon, 2006, BMC Bioinformatics; Cawley & Talbot, 2010, Journal of Machine Learning Research

Many splits of the claims, not one

One split can be lucky. The spread across many splits is what a design is judged on before it earns a test. Dietterich, 1998, Neural Computation; Bouthillier and others, 2021, MLSys

A frozen design and a single test

Choices made after seeing a result inflate it, and a test set that is used again leaks into the design. So each version is frozen before its seed is drawn and tested once. Nosek and others, 2018, PNAS; Dwork and others, 2015, Science

The honest range around accuracy

The Wilson interval stays reliable near perfect accuracy, where the textbook interval does not. The stricter test judges the bottom of that range. Brown, Cai & DasGupta, 2001, Statistical Science

Money as well as accuracy

Counting mistakes alone misleads when mistakes differ in cost: W2 is right almost every time and still loses money. Elkan, 2001, IJCAI

W3: certainty scaled to the money at stake

A textbook decision with the option to refer: act alone only when the expected cost of being wrong is small, and otherwise hand the claim to a person. Chow, 1970, IEEE Transactions on Information Theory; Correa Bahnsen, Aouada & Ottersten, 2015, Expert Systems with Applications

Where it cuts against a step

The quick confirmation

A review of more than a hundred experimental studies found that, on decision tasks, a person working with an AI did worse on average than the better of the two alone, most of all when the automated system alone was the stronger. A person approving a prepared file has that shape. Its saving stays a hypothesis until reviewers are tested with planted wrong files. Vaccaro, Almaatouq & Malone, 2024, Nature Human Behaviour

The second opinion

Combining two deciders pays when they are diverse, when their mistakes differ. When two language models both err, they often give the same wrong answer, and more accurate models more so, and this one also reads the decision model's findings. A second vote on the same facts is weak evidence: the certainty floor does the work, and the lab keeps it as a negative result. A language model is worth its cost where it reads or checks something the decision model cannot. Steyvers, Tejeda, Kerrigan & Smyth, 2022, PNAS; Kim, Garg, Peng & Garg, 2025, ICML

The limit that matters most

Reproduced, not yet carried over

The tests on fresh claims show that the result reproduces on new claims from the same generator. Whether it carries over to real claims is a separate question, and reporting guidance expects both to be stated. Nothing here tests the second yet. Justice, Covinsky & Berlin, 1999, Annals of Internal Medicine; Collins and others, 2024, TRIPOD+AI, BMJ

10How far to trust each number

What was tested, and how

1,000 synthetic claims, answered by ~typesafe/jev-latest through jev-openrouter. 5-fold cross-validation: every claim is scored by a workflow whose cut-offs and weights were fitted on the other four folds.

The claims are generated by code from a fixed seed, across auto, home, life and commercial insurance; the generator sets each claim's right answer and plants the frauds, manipulation attempts and missing papers.

Disclosure: the lab has no relationship with TypeSafe, the company that makes Jev. Jev was used through OpenRouter, and the lab's owner paid for every model call, Jev's and the language models', personally. The lab's owner designed, ran and judged every test, working with AI coding assistants; nothing here has yet been replicated independently.

Evidence: run 6827826e-b962-4bd8-bdb3-b163712fd660, exported from build 7c88683e4bce, dataset 96f9d5e30b51. Synthetic claims and stated assumptions: a demonstration, not a customer deployment.

How each number is known, strongest first

• Held-out testA frozen version run once on fresh claims it never saw, under a seed drawn when the run started. The strongest evidence here, and the only kind that can pass or fail the design.W3, 2 frozen versions
• Tested with real callsA real second model was called on the claims it would see.+2nd
• Measured in the engineThe rules as first built, run once over every claim. Measured differently from the later steps, so part of the jump to W1 is the method.W0
• Cross-validatedEach claim is graded by settings tuned on the other claims, never on itself.W1, W2, W3
• Replayed, assumed minutesThe stored answers routed through the new process for real; the minutes each new task takes are assumed.+QC, +Fraud, +Audit
• Simulated repliesCustomer replies written by a language model, including deliberately misleading ones.+Ask
• Assumed starting pointThe by-hand process, priced with the lab's assumed minutes. Nothing is measured.By hand

The evidence behind every figure on this site is published as it was exported, byte for byte: the manifest names each file and its hash.