Cross-lingual Calibration of Pre-Generation Success Probes for Multilingual LLM Routing

⚡ In one sentence AI-generated · check the original

Key findings

• English-trained probes retain useful cross-lingual discrimination but become less well calibrated after transfer.
• Pooled multilingual supervision improves both discrimination and calibration, yielding more reliable success estimates.
• The pooled router achieves 0.7% higher test success rate and 13.0% lower modeled cost versus always selecting the highest-average-success model, across 3,000 MATH problems in 10 languages and 8 model configurations.

📄 Abstract

Pre-generation success probes estimate response correctness from a language model's hidden activations before decoding, enabling cost-aware routing. While prior work has demonstrated their utility primarily on English inputs, we study their reliability across languages along three dimensions: (1) whether they preserve the ranking of likely successes and failures (DISCRIMINATION); (2) whether they retain probabilities that match observed success frequencies (CALIBRATION); and (3) whether they produce scores comparable enough across candidate models for cost-aware multilingual routing (UTILITY). Using 3,000 MATH problems in 10 languages and 8 open-weight model configurations, we compare cross-lingual transfer from English-trained probes and equal-budget pooled multilingual probes. English-trained probes retain useful cross-lingual discrimination but become less well calibrated after transfer. Pooled multilingual supervision improves both properties and yields more reliable estimates of success. In routing experiments, the pooled router achieves a 0.7% higher test success rate while reducing modeled cost by 13.0% relative to always selecting the model with the highest average success. These results show that multilingual routing requires success estimates that remain well calibrated and comparable across languages and models.