DigestAI news desk
hot OpenAI board member warns company is not on track to prevent catastrophic AI loss of control new R2VC Boosts Fact‑Checking Accuracy by 13.74% on FEVER Using Modular Retrieval and Calibration hot OpenAI Unveils GPT‑6 Astra: Record‑Breaking 3D Rendering, Loop‑Transformer Architecture new Study Finds Deictic Ambiguity Can Undermine Draft‑Verify‑Revise LLM Pipelines hot Mistral AI Secures $3.5B Series D, Valuation Tops $24B, Emphasizes Enterprise Control new New GLARE model improves meeting continuation forecasting on MDFB benchmark hot OpenAI delays IPO citing security risks after model escapes test environment new Chopthin-Consensus Power Sampling Boosts LLM Reasoning Accuracy Without Retraining hot OpenAI pledges to match Anthropic's embedded evaluator safety commitment new Occamy-1.0 Introduces Cost‑Efficient 35B Co‑Work Agent Model hot OpenAI launches Agents API in public beta, exposing Codex infrastructure to developers new Context-Augmented KG Training Boosts Multi-Hop QA Accuracy on Disease Graphs hot Meta launches Muse, a personal AI agent for everyday tasks on iOS and Android new Thermodynamical Genetic Algorithm Boosts LLM Artifact Efficiency by 29% on Bin‑Packing hot Anthropic reveals Houthis used Claude for ballistic missile development new HarnessDev Finds LLM‑Built Agent Harnesses Strong in Writing, Weak in Code Tasks
Research updated

Competence-Gated Pooling Improves Hybrid Event Forecasts Over External Signals

A new study introduces a "competence gate" that decides when a language model should be combined with existing market, crowd, or statistical forecasts for binary event prediction. By estimating domain‑level source weights from resolved outcomes and shrinking uncertain estimates toward a global baseline, the gate recalibrates pooled forecasts and boosts performance.

1 source primary source

Key points

  • Competence gate reduces Brier score from 0.0771 to 0.0732 across 2,357 binary questions
  • Four Qwen models and other LMs tested; gate outperforms global pooling and leakage‑safe priors
  • Verbal confidence fails to predict model usefulness; outcome‑based competence guides abstention

Evaluated on 2,357 resolved questions using five language models—including four Qwen variants—the method lowered the Brier score from 0.0771 to 0.0732, outperforming simple global pooling. The improvement held under strict leakage controls and against a time‑series prior, though it offered no gain on the ForecastBench market subset, where the gate largely deferred to the market. The authors also show that verbal confidence from the models is a poor indicator of competence, while outcome‑based estimates enable better abstention decisions.

The work provides a practical framework for selectively deploying language models only when they add measurable marginal value, helping forecasters avoid over‑reliance on AI predictions that may not improve existing signals.

Read the original at arXiv cs.AI · by Aditi Tiwari, Aashrith Bandaru, Heng Ji primary source Open source ↗
Topics · follow one to build your own front page
Qwen

The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us.

Comments

via GitHub Discussions

More in Research

All →

Related stories