Benchmarks

Ai2 has released StateBench, a four-level benchmark designed to evaluate how different language model architectures track evolving information across long contexts. Experiments show that while frontier and open pretrained models struggle with complex state updates over long inputs, recurrent and hybrid architectures like Gated DeltaNet generalize significantly better than transformers when trained directly on the tasks.

In an interview on the Latent Space podcast, Legora CTO Jacob Lauritzen discusses building AI agents for the legal industry following the company's $550 million Series E at an $8.8 billion valuation. He details how Legora evaluates models using simulated environments when customer data is strictly confidential, structures its product around core primitives like Tabular Review, and employs both legal engineers and forward-deployed engineers.

Architect Stephen Gallagher evaluated 13 AI models on their ability to generate technical architectural drawings directly from rough concept sketches and reference files. He found that Anthropic's Claude Opus 5.5 can produce permit-level construction sets and detailed blueprints by writing over 15,000 lines of Python code to generate vector linework and iteratively inspecting its own visual drafts.

This edition of the Import AI newsletter covers Realtime-Bench, a new benchmark showing that frontier language models struggle significantly with low-latency, real-time tasks unless time is paused. It also highlights Google DeepMind's Genie 4 world model, a neuro-symbolic reasoning position paper from startup Metis, and a Defense Intelligence Agency report on space warfare.

OpenAI has introduced a new benchmark called LifeVal to evaluate how effectively AI assists people with everyday tasks. The evaluation covers practical activities such as finding childcare, managing personal finances, and handling government paperwork.

SemiAnalysis has introduced GDPx, a new benchmark developed in collaboration with 14 specialized data partners to evaluate the agentic capabilities of frontier AI models across 11 high-value domains, including chip design, cybersecurity, and finance. Unlike previous static evaluations like OpenAI's GDPval, GDPx utilizes complex, realistic environments and currently sees top models scoring no higher than 12%.

Developer Nolan Lawson created a benchmark to evaluate how efficiently AI coding agents build the same web application across twelve different JavaScript frameworks. Testing with Claude Code revealed significant variance in token usage, cost, and build time, with lightweight setups like Hono+htmx and Astro proving faster and cheaper for agents than Angular or SvelteKit.

A developer successfully got the GPT-5.6 Sol model, running in Codex, to autonomously beat the factory-building game Factorio on peaceful mode in 13.5 hours of game time. Interacting via a Python library under human-like game restrictions, the model improved upon GPT-5.5 by using transport belts and ultimately launched a rocket, despite struggling with late-game scaling.

MIT researchers have launched the LLM Election Observatory, a public dashboard that tracks in real time how popular AI chatbots respond to questions about the 2026 U.S. midterm elections. By periodically running a fixed set of prompts with varying user locations and political affiliations, the project documents how chatbot framing, cited sources, and tone shift over time and across different user profiles.

Anthropic hiring manager Felix Reiseberg announced he has stopped using traditional coding interviews because Claude now outperforms humans on standard coding challenges. The daily roundup also covers Google DeepMind's release of the Genie 4 world model, major mathematical formalization milestones by Axiom, OpenAI, and Anthropic, and updates to the Artificial Analysis Intelligence Index v5 and FrontierCode 2.0 benchmarks.

A Wired journalist experimented with using multiple AI agents to manage a Premier League fantasy soccer team, finding that while multi-agent setups improved suggestions, the system still failed to beat the average human score. Researchers testing frontier models on prediction markets and benchmarks like Prophet Arena and Prediction Arena similarly note that AI forecasting is improving rapidly, though models still generally lag behind top human superforecasters and lose money in live trading.

Surge AI introduced DAYJOB: Tax, a benchmark evaluating AI agents across 50 realistic, multi-file professional tax assignments created by tax experts. The benchmark tests end-to-end professional judgment and messy record reconciliation, with top models achieving at most a 10.8% pass rate.

Artificial Analysis has updated its Intelligence Index to v5, dividing weights equally across knowledge work, coding, science, and general tasks while shifting toward private benchmarks. The commentary analyzes how these updates widen the apparent gap between closed and open-weight models and discusses the trade-offs of relying on single aggregate metrics.