StakeBench: Evaluating Language Understanding Grounded in Market Commitment
StakeBench: 评估基于市场承诺的语言理解
Yunhua Pei, Jingyu Hu, Yiwei Shi, Hongnan Ma, Weiru Liu, John Cartlidge
AI总结 提出StakeBench框架,通过将市场评论与可验证的交易记录关联,从市场行为中自动生成监督信号,评估语言模型对市场承诺的理解能力。
详情
- Comments
- 21 pages, 2 figures, 20 tables. Preprint. Dataset and evaluation code included
现有的金融自然语言处理基准通常依赖外部观察者提供的标签,衡量语言如何被感知而非说话者在市场中承诺了什么。我们引入StakeBench,一个基于市场承诺的语言理解评估框架。StakeBench将来自2261个已结算市场的560,876条评论与Polymarket和Manifold上可验证的头寸、行动和市场赔率记录相关联。监督信号来自可观察的市场行为。头寸方向、评论后交易行动和市场赔率轨迹取代了人工标注。四个诊断任务测试模型是否检测到市场承诺、识别揭示的方向、预测未来行动以及执行集体赔率预测。三个承诺感知指标衡量与揭示偏好而非感知情绪的一致性。有效性审计和明确的解释边界有助于区分可观察的承诺信号与潜在信念和因果市场赔率影响。在15个LLM、18个主题和平台设置中,模型部分恢复了头寸方向信号,定向准确率从0.506到0.599,但在后续任务中出现结构性失败。15个模型中有10个在未来行动预测中崩溃为一到两个行动标签,且没有模型在集体赔率预测中持续优于朴素赔率方向基线。模型规模与性能不相关,金融领域微调不改善揭示方向识别,平台激励强烈影响高阶结果。StakeBench在CC-BY 4.0许可下附带评估代码和数据集。
Existing financial NLP benchmarks often rely on labels supplied by outside observers, measuring how language is perceived rather than what speakers have committed to in the market. We introduce StakeBench, an evaluation framework for language understanding grounded in market commitment. StakeBench links 560,876 comments from 2,261 resolved markets to verified position, action, and market-odds records across Polymarket and Manifold. Supervision is derived from observable market behavior. Position sides, post-comment trading actions, and market-odds trajectories replace human annotation. Four diagnostic tasks test whether models detect market commitment, identify the revealed side, anticipate future action, and perform collective odds projection. Three commitment-aware metrics measure alignment with revealed preferences rather than perceived sentiment. Validity audits and explicit interpretation boundaries help distinguish observable commitment signals from latent belief and causal market-odds impact. Across 15 LLMs and 18 topics and platform settings, models partially recover position-side signals, with Directed Accuracy from 0.506 to 0.599, but show structural failures on later tasks. Ten of the fifteen models collapse to one or two action labels in future action anticipation, and no model consistently improves on the naive odds-direction baseline in collective odds projection. Model scale is not correlated with performance, finance-domain tuning does not improve revealed-side identification, and platform incentives strongly shape higher-order results. StakeBench is packaged with evaluation code and dataset under CC-BY 4.0.