TypeSafe AI's Jev returns typed judgments instead of text: give it some state, ask a choice, score or yes/no question, and get an answer with probabilities in a few hundred milliseconds. For routing and classification, that is a very useful primitive. In regulated work we also need something else: the same input should produce the same answer, every time. So we checked.
What TypeSafe says
Jev's FAQ answers the question directly. It defines determinism as returning the same result for an identical input, calls it less valuable than consistency (similar decisions when the meaning stays similar, even if the wording changes), and states: “Jev is designed for consistency.” So Jev does not promise determinism, and we are not testing a claim it makes. We wanted to know what that design choice means in practice.
TypeSafe's consistency cookbooks send the same content to Jev 15 times and report label flips on 2 of 8 questions. Each of those calls, however, adds a fresh uid field, and the authors note that this cannot separate sensitivity to an irrelevant field from variation on identical requests. We removed that ambiguity.
Setup
- Data. ContractNLI (Koreeda & Manning, 2021; CC BY 4.0), a natural-language-inference benchmark over non-disclosure agreements. First 10 contracts of the dev split × 2 hypotheses (nda-2: “Confidential Information shall only include technical information”; nda-7: “Receiving Party may share some Confidential Information with some third-parties”) = 20 questions.
- Request. The full contract as
state; onechoicequestion with three options (Entailment, Contradiction, NotMentioned), each with a one-line definition. - Model. Pinned to
jev-1.13.0. All 600 responses reportedjev-1.13.0. - Calls. For each question we built the request body once and sent the identical bytes 30 times: 15 sequential calls, then 15 concurrent calls. One day (2026-10-01), from a VM in us-central1.
Results
| Identical across all 30 calls | Questions |
|---|---|
| Chosen label | 18 / 20 |
| Probabilities (as returned, two decimals) | 6 / 20 |
| Probabilities, sequential calls only | 6 / 20 |
| Probabilities, concurrent calls only | 6 / 20 |
- Two labels flipped. Contract 3 / nda-7: Entailment 25 times, Contradiction 5 times. Contract 15 / nda-2: Contradiction 27 times, NotMentioned 3 times.
- Probabilities moved on 14 of 20 questions. The spread of a single option's probability across 30 calls had a median of 0.04 and a maximum of 0.15.
- Not a load effect. Sequential calls varied as much as concurrent ones.
- Fast. Median latency was 0.22 s.
Why it matters
We did not measure accuracy, so nothing here says Jev is wrong, and Jev behaves as its FAQ describes. We agree that consistency matters. But in regulated assessment, consistency and determinism are not substitutes. A quality or compliance reviewer must be able to rerun an assessment and get exactly the result that was recorded, because validation, audit trails and investigations depend on it. If the same input can be routed one way today and another way on the next run, the recorded result cannot be reproduced for an auditor, however consistent the system is on average.
Determinism is engineerable, end to end
At GFLOPS we are building AskDona Assessment, a system that is deterministic end to end: not only the language model, but every stage from the source documents to the final assessment.
We are applying this end-to-end deterministic system to assessment work in regulated industries, where reproducing a recorded result exactly is part of the job.
Limitations
- A small sample: 20 questions, one model version, one day, one region.
- Jev returns probabilities rounded to two decimals, so smaller variation is invisible to us.
GFLOPS, 2026-10-01. Data: ContractNLI (Koreeda & Manning, Findings of EMNLP 2021), CC BY 4.0. Our script and raw responses are available on request.