JevCascade: Readout Stability and Inference-Time Compute Allocation for Prefill-Only Decision Models

⚡ In one sentence AI-generated · check the original

Key findings

• Menu-only interventions are predicted to within 4.2 points across six model families, sixteen datasets and two task types, exactly for one family; a probability-level variant errs by 21.0 points.
• Uniform extra passes buy calibration but almost no accuracy; at matched cost a confidence cascade beats every scheme re-asking the same model.
• A 0.8B model on a curated 5-candidate menu reaches 95.4% on CLINC150 versus 80.0% for a 4B model on the full 150-label menu.

📄 Abstract

We present JevCascade, an inference-time compute allocation framework for prefill-only decision models, built on readout stability and zero-label restricted readout prediction. Prefill-only decision models score every candidate in a menu in a single forward pass and never decode, which makes one call one to two orders of magnitude cheaper than a same-scale generative language model. We show that this read-out structure carries a testable property. When an intervention changes only the menu and leaves the input text fixed, the post-intervention accuracy is already determined by the cached first-pass distribution. The restriction predictor restricts the pass-1 probabilities to the menu, renormalizes and reads off the argmax, with no labels and no second pass. Across six model families, ten core datasets (sixteen in total) and two task types, menu-only interventions are predicted to within 4.2 points, exactly for one family. The property is ordinal, not probabilistic, since a probability-level variant of the same estimator errs by 21.0 points, and class-specific, since same-scale generative models err by 1.6 to 15.8 points and degrade as they grow. It makes inference-time compute decidable before deployment. Uniform extra passes buy calibration but almost no accuracy; at matched cost a confidence cascade beats every scheme that re-asks the same model, and curating the menu beats enlarging the model, a 0.8B model on a curated 5-candidate menu reaching 95.4% on CLINC150 against 80.0% for a 4B model on the full 150-label menu.