Jev-ToM: Evaluating Theory of Mind in Typed Decision Models

⚡ In one sentence AI-generated · check the original

Key findings

• TypeSafe Jev reaches 73.2% source-balanced accuracy including native controls, ranking first or second across all six capabilities.
• Jev was compared against five typed decision models and seven generative baselines on 11,117 English questions from nine ToM benchmarks.
• Confidence-based selection remains benchmark-dependent, and related-question cues account for most of Jev's downstream gains.

📄 Abstract

Dialogue and collaborative assistants repeatedly judge what a partner knows, believes, or intends. Typed decision models offer a potential alternative to generative large language models for bounded Theory of Mind (ToM) selection. We present Jev-ToM, comparing five typed decision models and seven generative baselines on 11,117 English text questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy including native controls and ranks first or second across all six capabilities, leading beliefs, goals/desires/intentions, and emotions. Other typed models show distinct strengths across tasks. Separate resource comparisons suggest that Jev could reduce the cost and waiting time of repeated ToM selection. Confidence-based selection remains benchmark-dependent, and supplied reasons can repair errors or overturn correct answers depending on their attachment. Candidate expansion reveals distinct selection and output-validity problems. Related-question cues account for most of Jev's downstream gains, with task-dependent effects from predicted states. The findings motivate testing Jev as a ToM selection component in assistants, with task-specific confidence calibration, candidate design, and output validation as integration priorities.