Can code handle routine pharmacy decisions, with a narrow model answering?

Demo, not validated · not clinical advice

A drug maker must send serious, unexpected adverse events to the regulator within fifteen days. Someone reads every incoming report to decide. The demo asks whether Jev can flag the serious ones in real public reports (VAERS), so a person starts the clock sooner.

How it works

The result, at the built line

Settled with no person: none. Every report still reaches a person, in the safety queue or the routine one; the demo only sorts which a person reads first. Any gain would be speed, starting the clock sooner, and that is not measured.

How far reading the label alone gets you

Some narratives state the sender’s own seriousness verdict: 83 of 300. Reading that label alone would catch 21 of the 100 serious reports; Jev caught 80, including 21 of those 21. The label is no answer key either: it disagrees with the reporter’s coding on 20 of the 83. The other 217 narratives carry no label (76 of them coded serious), and that is where all 20 misses fall.

Demo, not validated · not clinical advice

A pharmacist or payer reviewer reads each request against the drug’s criteria. Most meet them. The demo asks whether code can approve those on its own, from Jev’s answers, and send everything else to a person. It never denies.

How it works

Every run, failures included

Hatched: that run also approved requests it should not have. Each run is scored under its own rule.

Work taken off a person, and what the model cost

What made the difference: the first run’s notes, one change at a time

On the same notes, the gentler rule alone approves 40 of 178 complete requests, and adding the required-items list takes it to 76; the list removes the false “item missing” answers. The request date alone changes little: 43 against 40 under the gentler rule, and nothing under the other two. Drug-level differences from the gentler rule’s own run (on new requests, 78 of 178) are not separated from its new seed and notes.

In the last design, 15 of 15 contradictions went to a pharmacist. Adalimumab, the dearest drug, still got none of its 24 complete requests approved: its cost raises the line above what Jev’s answers reached. The wrong approvals in the gentler rule’s own run, and under it on the first run’s notes, came from a fault in the answer key, a “contradiction” that wasn’t one; the last design’s generator fixes it.

Demo, not validated · not clinical advice

Clinics renew routine prescriptions under standing orders: labs current, a recent visit, no new problem. The demo asks whether code can approve those from Jev’s reading of the chart note, while its own rules, from the fill history alone, send controlled drugs, early refills and high-alert drugs to a pharmacist.

How it works

Three designs, the same answers

Each design routes the same requests, the same notes and the same Jev answers; only code’s rules change, and nothing was asked again. As built, Jev judged the lab and visit dates. The second design makes two changes: code reads the dates from structured records, and Jev’s two date answers leave the approval decision. The third also leaves Jev’s manipulation answer out of the approval decision, while still quarantining a clear attempt. Here the dates code reads are the generator’s own facts, the ones the answer key is built from, so the date check is right by construction; real records can be missing, late or wrong, and the gain would be smaller.

The trade-off. As built: 13 wrong approvals, all overdue refills. Code checks the dates: none wrong, and 45 of 45 requests needing labs or a visit sent back, but only 20 of 110 approved. Manipulation answer out of approval: 104 of 110 approved, and 1 of the 9 manipulation attempts approved (1 of the 6 reworded ones).

Where each gain comes from, computed from the stored answers: the date check alone removes every wrong approval (0 of 16) and leaves approvals at 16 of 110; dropping Jev’s two date answers from the approval decision is what takes them to 20.

Jev’s manipulation answer does not separate clean notes from reworded attempts at the low end. Under the third design, approved requests scored up to 0.49 on it; the reworded attempt it approved scored 0.36, and 42 clean requests it approved scored at least as much. No line on that answer alone could have stopped the attempt without also holding back those clean requests.

Both re-routes were chosen after seeing earlier results on these same requests: the second after the first run, the third after the second. Their figures are optimistic, and neither has been tested on new requests.

As built, in detail

As built, code isn’t safe to approve refills on its own here: of its 29 automatic approvals, 13 were wrong (44.8%, range 28.4 to 62.5%), every one an overdue refill.

What worked

Every request, by what was planted

All 300 requests were answered. In the first phase the note check refused 22 notes: 16 as too short, under a floor since lowered, and 6 manipulation attempts the writer had reworded in its own voice, which the check did not yet recognise. All 22 passed the corrected check in phase two and were asked; the reworded attempts are the 6 counted under manipulation.

89 requests went to a pharmacist on code’s own fill rules, whatever Jev answered: controlled drugs, early refills and high-alert drugs. Jev never sees the fill data.

• Real cases with a checked answer key, not synthetic ones or a reporter’s own coding.
• Each drug’s required items in what the model sees, and a line for each answer rather than the weakest of all.
• Date checks in code: lab and visit dates read from structured records, as the fill rules already are, not from a note.
• An independent check of each note for instruction-like text, or a person, before any automatic approval.
• A held-out test on cases no setting was tuned on, before any line is used.
• A person on every case code doesn’t settle, with their time measured.

Every figure is from a single run on a few hundred cases, and every rate carries its ninety-five per cent range (a Wilson interval). Where a setting was chosen after seeing a run, the page says so and its figures are optimistic. The predictions were written before each run and scored by the lab’s research session; some were informed by earlier runs or by sample-level information, and each is marked in its record.

The research session’s score across the pharmacy predictions: 20 hits of 27 scored, all marked informed in the record.

Where each figure comes from

Each run’s full export is held in the lab’s private repository, and this page is computed from a frozen extract of it with no report or request text. Every route on the page is re-derived from Jev’s answers and the run’s own rule constants when the page is built, and the build fails if one differs. The synthetic notes were written by a language model, named in the table; the VAERS reports are public.

Disclosure: the lab has no relationship with TypeSafe, the company that makes Jev. Jev was used through OpenRouter, and the lab’s owner paid for every model call, Jev’s and the language models’, personally. The lab’s owner designed, ran and judged every demo, working with AI coding assistants; nothing here has been replicated independently.