Worth Knowing
All topics

Topic

Benchmarks

AI news about evaluations, benchmarks and comparisons of models.

7 stories in the last 3 days

Yesterday

2 stories
  1. Worth knowing Ai2

    StateBench: Testing how language models track state changes over long documents

    This research empirically demonstrates the theoretical limitations of standard transformers in long-horizon state tracking and highlights the potential of recurrent and hybrid architectures to overcome them.

    Ai2 has released StateBench, a four-level benchmark designed to evaluate how different language model architectures track evolving information across long contexts. Experiments show that while frontier and open pretrained models struggle with complex state updates over long inputs, recurrent and hybrid architectures like Gated DeltaNet generalize significantly better than transformers when trained directly on the tasks.

    2 sources

Mon, Nov 2

5 stories
  1. Worth knowing Latent Space

    Building AI for Lawyers - Jacob Lauritzen, CTO of Legora

    It offers a detailed look into the engineering, evaluation, and product architecture required to scale an enterprise AI agent company in a strict, high-stakes domain where customer traces cannot be inspected.

    In an interview on the Latent Space podcast, Legora CTO Jacob Lauritzen discusses building AI agents for the legal industry following the company's $550 million Series E at an $8.8 billion valuation. He details how Legora evaluates models using simulated environments when customer data is strictly confidential, structures its product around core primitives like Tabular Review, and employs both legal engineers and forward-deployed engineers.

    1 source
  2. Worth knowing Hacker News

    Claude Opus 5.5 Can Draw Permit-Level Architectural Drawings

    This demonstrates how frontier models combining coding, vision, and long-horizon agentic loops are expanding into complex technical domains like CAD and architectural drafting that previously resisted AI automation.

    Architect Stephen Gallagher evaluated 13 AI models on their ability to generate technical architectural drawings directly from rough concept sketches and reference files. He found that Anthropic's Claude Opus 5.5 can produce permit-level construction sets and detailed blueprints by writing over 15,000 lines of Python code to generate vector linework and iteratively inspecting its own visual drafts.

    1 source
  3. Worth knowing Import AI

    Import AI 479: Spacewar; low-latency tasks versus LLMs; Genie 4

    It synthesizes notable developments in world models, neuro-symbolic architectures, and real-time agent evaluation, pinpointing where current frontier models fail under real-world latency constraints.

    This edition of the Import AI newsletter covers Realtime-Bench, a new benchmark showing that frontier language models struggle significantly with low-latency, real-time tasks unless time is paused. It also highlights Google DeepMind's Genie 4 world model, a neuro-symbolic reasoning position paper from startup Metis, and a Defense Intelligence Agency report on space warfare.

    1 source
  4. Worth knowing OpenAI

    LifeVal

    A new benchmark from a major frontier lab focused on everyday consumer tasks signals an increasing push to evaluate AI models on practical, real-world utility rather than just academic and coding proficiency.

    OpenAI has introduced a new benchmark called LifeVal to evaluate how effectively AI assists people with everyday tasks. The evaluation covers practical activities such as finding childcare, managing personal finances, and handling government paperwork.

    1 source
  5. Worth knowing SemiAnalysis

    GDPx: Measuring Frontier Model Capabilities Across the Most Valuable Domains

    GDPx addresses the saturation and unrealistic constraints of existing knowledge-work benchmarks by testing AI agents within rich, interactive work environments across economically critical sectors.

    SemiAnalysis has introduced GDPx, a new benchmark developed in collaboration with 14 specialized data partners to evaluate the agentic capabilities of frontier AI models across 11 high-value domains, including chip design, cybersecurity, and finance. Unlike previous static evaluations like OpenAI's GDPval, GDPx utilizes complex, realistic environments and currently sees top models scoring no higher than 12%.

    1 source