Teaching an open model to say how sure it is

01 · What we were trying to find out

A lot of what language models do inside software isn’t writing. It’s deciding. Is this ticket urgent? Which team gets it? Is this shell command safe to run? For decisions like these you don’t want a paragraph back. You want a probability, and you want that probability to mean something: when the model says 80%, it should be right about 80% of the time.

In September, TypeSafe launched Jev, a closed “System One” model built for exactly this. You send it a piece of state and a set of typed questions: yes/no, choice (pick one of up to 255 options) or score (a 2-to-10-level rating). It answers all of them at once with probabilities instead of text. It’s fast and cheap, and it’s pitched as calibrated. Independent tests have been less kind to that last claim. One study measured a calibration error of 0.107 on synthetic support tickets, about four times its noise floor. TypeSafe’s own docs show a question and its negation getting probabilities that add up to 1.19.

We didn’t benchmark Jev itself; that’s for a later post, once we have an account. Jev set the question:

Can an open model on a single consumer GPU return typed, calibrated probabilities in one forward pass, and does that calibration hold on kinds of tasks it never trained on?

02 · The setup

The readout: probabilities without generating anything

The core trick is simple. For each question we build a prompt: the state, the question, and the options, each tagged with a one-token label (A, B, C… for choices, 0–9 for rating levels, yes/no for yes/no). Then we run one forward pass and look at the model’s next-token distribution at the very last position. The probability mass on each label is the answer. We renormalize over the labels, and that’s the distribution we report.

Nothing is decoded, so there’s no sampling and no parsing of free text, and a 77-option question costs the same single pass as a yes/no one. It also means we measure the model’s own probabilities rather than probabilities it writes out as text, and that turns out to matter a lot (§4).

One architectural wrinkle changed the plan. The backbone family, Qwen3.5, is a hybrid: three of every four layers are Gated DeltaNet linear attention, which carries a single running state along the sequence rather than an attention matrix. The usual way to ask many questions about one state in a single packed sequence is a tree attention mask, and that simply doesn’t exist for those layers. So each question gets its own pass for now. Sharing the encoded state across questions is an optimization for later.

What we measured on

We built an eval set from public datasets, grouped into seven task families:

Family	Source	Question type	Seen in training?
Entailment	MultiNLI	choice + yes/no	yes
Reading comprehension	RACE	choice (4)	yes
News topic	AG News	choice (4)	yes
Star ratings	Yelp reviews	score (5 levels)	yes
Negation pairs	BoolQ	two yes/no	yes
Banking intent	Banking77	choice (77)	never
Offensive tweets	TweetEval	yes/no	never

That’s 2,701 questions. The two bold families are the real test. Nothing like them appears in training: no intent classification and no toxicity data of any kind. Getting calibration right on familiar tasks is the easy half; the question is whether it transfers.

What we trained

One LoRA epoch (rank 16) on 29,001 questions from the training splits of nine families: the five in-distribution ones, plus DBpedia topics with 2-to-14 random options, science multiple choice, Amazon star ratings and sentiment pairs. The loss is a sum of proper scoring rules on the readout’s own distribution: log loss, a Brier term, a ranked-probability term for ratings, and a penalty whenever a question and its negation don’t add up to 1. Proper scoring rules are the standard tool for this job, because the loss is minimized exactly when the stated probabilities match reality.

Then we ran that identical pipeline on five models: Qwen3.5 at 0.8B, 2B, 4B and 9B, and Qwen3.8 at 27B.

03 · How we measured it

Accuracy is the plain fraction of questions where the top answer was right.

Calibration error (ECE) is the headline number. Group every answer by the model’s confidence in its top choice, 15 bins from 0 to 1, and within each bin compare the average confidence to the actual accuracy. ECE is the weighted average of those gaps:

$$\text{ECE} = \sum_{b} \frac{n_b}{N}\,\bigl|\,\text{conf}_b - \text{acc}_b\,\bigr|$$

An ECE of 0.05 means the model’s stated confidence is off by about five percentage points on average. One caveat matters a lot here: with only 300 questions per family, even a perfectly calibrated model shows an ECE of about 0.02–0.06, just from sampling noise. We report that noise floor next to every number, and anything near it is indistinguishable from perfect.

The easiest way to see calibration is a reliability curve: stated confidence on the x-axis, actual accuracy on the y-axis. A calibrated model sits on the diagonal. A curve that sags below the diagonal belongs to an overconfident model.

Negation error checks internal consistency. We ask “is the answer yes?” and “is the answer no?” about the same passage, and measure how far the two probabilities are from summing to 1. The target is under 0.05.

04 · Two baselines, and a surprise

The obvious baseline is to ask a strong model for its probabilities. We prompted a 27B model to write them out as JSON. It was accurate, and overconfident on every question type, with calibration errors of 0.06–0.11. Its probabilities were also coarse: it answered 0.95 or 0.05 almost every time. A model writing a number is telling you what a confident answer sounds like, not reading its own uncertainty.

The single-pass readout, before any training, was already better calibrated than that on yes/no and choice questions (ECE 0.03–0.06 on the 9B), just by reading the probabilities the model already has. Its weak spots were accuracy, rating questions (badly overconfident) and consistency. On negation pairs it had a yes-bias: “is the answer yes?” and “is the answer no?” both leaned yes, giving a negation error of 0.21.

The surprise concerned which checkpoint to start from. A common argument says the chat-tuned version of a model has had its calibration damaged by preference training, so you should start from the raw pretrained “base” model. We tested both on the 9B, and the argument didn’t hold here: the chat-tuned model was better calibrated than the base model, and more accurate, with every question type pooled. Everything that follows starts from the chat-tuned models.

05 · What one LoRA epoch did, from 0.8B to 27B

Here’s the scorecard after training. Every cell is accuracy / calibration error; the 4B and 9B rows are the mean of two training seeds.

After LoRA	Yes/no	Choice	Rating	Negation error	Training time
0.8B	0.86 / 0.021	0.85 / 0.023	0.63 / 0.059	0.049	~12 min†
2B	0.90 / 0.016	0.89 / 0.027	0.62 / 0.072	0.038	16 min
4B	0.92 / 0.037	0.90 / 0.030	0.66 / 0.073	0.023	34 min
9B	0.93 / 0.028	0.91 / 0.023	0.68 / 0.046	0.024	51 min
27B	0.94 / 0.018	0.93 / 0.021	0.70 / 0.051	0.019	152 min
27B, prompted	0.91 / 0.037	0.87 / 0.073	0.63 / 0.114	0.046	—

Trained task types only. Rating questions have 300 items, so their ECEs sit on a noise floor of about 0.06. Times are one epoch on RTX 5090s: the 0.8B–9B on two cards, the 27B split across two. †The 0.8B’s run wasn’t timed; the estimate is from its logged throughput.

Three findings stand out.

1. On familiar tasks, the recipe does most of the work

Before training, choice accuracy ran from 0.59 (0.8B) to 0.87 (27B). After one epoch, every model from 2B up lands between 0.89 and 0.93, with calibration errors around 0.02–0.04, mostly at the noise floor. Rating questions go from near chance at the small end (0.21 for the 0.8B, against a chance level of 0.20) to 0.62–0.70 at every size. And the negation problem is solved everywhere: 0.16–0.41 before training, under 0.05 after, at every size.

Practical reading: if you know which kinds of decisions you need, a 2B model and a quarter hour of training get you calibrated, typed answers that match the 27B we prompted.

2. On unseen task types, there’s a size threshold

The 77-way banking intent task is where the sizes separate. The 0.8B and 2B are near chance both before and after training (0.04–0.21). The 4B, 9B and 27B reach 0.71–0.81. Training adds 2–9 points for those larger models, but it doesn’t create the ability in a model that lacks it. Somewhere between 2B and 4B, the model starts being able to read 77 option labels and pick the right one.

3. Training fixes overconfidence on unseen tasks, for the larger models

Before training, the 9B and 27B were confidently wrong on unseen tasks, with calibration errors of 0.13–0.19 on 77-way intent. One epoch on other tasks cut that roughly in half: 0.192 → 0.093 for the 9B and 0.133 → 0.062 for the 27B. For the 9B we trained twice with different seeds and got 0.093 and 0.092. The 27B’s result is a single run.

Nobody hit the goal, though. The target was under 0.05 on unseen task types, and the best result on the 77-way task is 0.062. That’s the open problem the rest of this project is about.

06 · The lessons that weren’t in the plan

A finding that didn’t survive a second seed

After the first round, the 4B looked like it had an interesting story: training made its calibration on unseen tasks worse (0.049 → 0.071 on tweets, 0.079 → 0.106 on intent). It fit a tidy narrative about mid-sized models over-fitting to their training tasks. We ran a second seed before believing it. Seed 2 went the other way: 0.049 → 0.033 and 0.079 → 0.076. The effect was noise.

Lesson: on 300-question eval sets, any calibration difference of about 0.03 or less needs at least two seeds before it counts as a finding. Its sign can flip with the next run. The 9B’s improvement, by contrast, reproduced to within 0.003. That’s what a real effect looks like at this sample size.

Two of our “findings” were measurement bugs

The first time we measured negation, the model said “yes, 95%” to both “is the answer yes?” and “is the answer no?”, which looks like a spectacular consistency failure. It was our prompt. We’d asked “is the answer no?” and then asked for “the probability that the answer is yes”, which is ambiguous. With explicit criteria on each question, the pairs came out as complements.

Separately, a numerical comparison made the fast GPU kernels for the linear-attention layers look wrong: probabilities differed by up to 0.05 from the reference implementation. In full precision they agreed to 0.001. The gap came entirely from bf16 arithmetic, the same for both implementations.

Lesson: when a result is dramatic, rule out the measurement first. Both of these would have made a good paragraph, and both were false.

The extra loss terms didn’t earn their keep

We trained the 9B a second time using only plain log loss, dropping the Brier, ranked-probability and consistency terms. It matched the full objective within noise on every metric, including negation consistency (0.022 both ways). The consistency penalty turned out to be redundant: the training data already contains both halves of every negation pair with correct labels, so ordinary log loss teaches the model that “no” is the complement of “yes”.

Four GPUs aren’t faster than two when they talk over PCIe

For the 27B we tried sharding the model across all four cards (FSDP). It trained correctly, matching our verified two-GPU path to within 0.003 on every loss term, but it ran at 1.0 items per second against 2.3 for simply splitting the model across two cards. Our cards talk through the host’s PCIe bridge, with no direct GPU-to-GPU link, and FSDP moves every layer’s weights between cards on every pass. We kept the two-card split.

07 · Try it

For 48 hours after this post goes up, the 4B model (with its second-seed adapter) is live on one of our GPUs. The 4B is the sweet spot of the study: nearly the 9B’s accuracy on familiar tasks, and it crosses the size threshold on unseen ones. Write some state (a support ticket, a request for an agent, any JSON blob), ask it typed questions, and watch the probability distributions come back. The request and response use the same shape as Jev’s API.

A few things to know. It answers in about a quarter of a second, with one forward pass per question and no text generated. It’s rate-limited and doesn’t store what you send, but don’t paste anything private. Choice questions are capped at 26 options, so the 77-way intent task won’t fit. And it’s the model from the table above, with the same strengths and blind spots: it’s good at the kinds of decisions it trained on, and less sure elsewhere.

08 · Where we go next

• More unseen task types, and more seeds. Two unseen families is a thin base for the transfer claims, and three of our five sizes were trained once. Both are cheap to fix, and both come before any stronger claim.
• Close the gap on unseen tasks. The 0.05 target isn’t met on the 77-way task. The obvious lever is a broader training mixture (more kinds of decisions, not more of the same), which is what the size study suggests matters.
• Separate size from generation. Our 27B is a newer model generation than the rest, so the top of the curve changes two things at once. A same-generation ~27B would untangle that.
• Measure Jev itself. Our evaluation harness already speaks Jev’s API. With an account, the head-to-head comparison is a single command, which is the comparison this whole project was set up to make.

---

All of this ran on a homelab box with four RTX 5090s. Numbers are from held-out evaluation; seeds and noise floors are noted where they matter. Built on the open Qwen3.5 and Qwen3.8 models, the flash-linear-attention kernels, and public datasets from the MultiNLI, RACE, AG News, Yelp, BoolQ, Banking77 and TweetEval authors. Thanks to all of them.