BTF-2 Benchmark Tests AI Forecasting Reasoning

🔑 Enhanced Key Takeaways

• •BTF-2 utilizes a 'frozen' corpus approach to solve the data contamination problem prevalent in LLM evaluation, ensuring that agents cannot access information published after the event date.
• •The benchmark introduces a novel 'Counterfactual Sensitivity' metric that measures how agents adjust their probability estimates when provided with specific, injected adversarial scenarios.
• •The research team behind BTF-2 identified that frontier models exhibit a 'confirmation bias' in forecasting, where they disproportionately weight evidence supporting their initial hypothesis over contradictory data.

🛠️ Technical Deep Dive

• •Corpus Architecture: Uses a static, time-stamped snapshot of 15 million documents (news, policy papers, financial reports) indexed via a vector database to simulate a 'closed-world' information environment.
• •Evaluation Engine: Implements a multi-stage pipeline that separates 'Information Retrieval' (RAG performance) from 'Probabilistic Reasoning' (Brier score calculation) to isolate failure points.
• •Agent Prompting: Employs a chain-of-thought (CoT) framework specifically tuned for 'Pre-Mortem' analysis, forcing agents to generate three distinct failure scenarios before outputting a final probability distribution.
• •Sensitivity Analysis: Uses a perturbation-based testing method where key variables in the prompt are modified to check for non-linear shifts in the agent's confidence intervals.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline