Image created with gemini-3.1-flash-image-preview with claude-opus-4.7. Image prompt: { “category”: “Benchmarks”, “short_title”: “The Scale of Judgment”, “concept_description”: “A stained-glass panel depicting a giant set of golden balance scales at center, one pan holding a smug, overfed housecat wearing a numbered leaderboard collar, the other pan holding a single perfect feather from the red-tailed hawk — and the feather outweighs the cat. Above, the hawk hovers as arbiter with a halo of ruled tick-marks like a measuring rod; below, the minister kneels in running gear, reading the verdict on the scales. The metaphor: benchmarks as moral weighing, where inflated scores get outweighed by a single honest measure of real-world consequence.”, “image_prompt”: “A Lutheran stained-glass panel of golden balance scales weighing a plump housecat with a numbered leaderboard collar against a single red-tailed hawk feather that tips the scale, a haloed hawk hovering above as arbiter with a ruler-of-tick-marks nimbus, a bald athletic minister in running gear kneeling below gazing up in reverent recognition, framed by gothic tracery in cobalt sapphire ruby amber and emerald jewel-tone glass with heavy black leading and backlit cathedral glow, a bold blackletter title-card banner reading BENCHMARKS across the base. }
It’s over Mythos is insane
https://x.com/scaling01/status/2055362921803211248
May 13, 2024: GPT-4o is released and scores below 10% on AIME 2024 (US math olympiad entry questions) Dec 20, 2024: o3-preview saturates AIME 2024 Jul 21, 2025: Gemini DeepThink scores IMO gold May 20, 2026: internal GPT models makes Erdös unit-distance problem breakthrough
https://x.com/scaling01/status/2057246143881609510
The full Frontier Risk Report includes more than we can describe here, with over 200 pages of appendix materials detailing how the pilot worked, what we asked participants, and what we learned, along with transcripts from our evals. See our website:
https://x.com/METR_Evals/status/2056800047258649049
Everyone has seen the @waitbutwhy cartoon of AI capability growth with a “”you are here”” indicator just before the exponential really starts, but the independent assessments of both METR and the UK’s AISA do seem to show that we are past that point now (until we hit a slowdown?)
https://x.com/emollick/status/2054781154553729088
RL roughly on trend, multimodality on trend, strange to see them report mediocre MRCR and ARC-AGI-2. Given the speed, it might well have fewer active parameters than Flash-3 (so they both shrink the batch and grow margin). Will be a successful model until we get some 5.6-Mini.
https://x.com/teortaxesTex/status/2056794752167645653
(1/n) One of the big challenges on our roadmap at @VibrantLabsAI is scaling agent benchmarks while maintaining a strong reward signal. Good verifiers are the bedrock of usable agent benchmarks. Bad verifiers can inflate model failure rates, and they often hide actual
https://x.com/Shahules786/status/2056773476585816255
💥Today we release InferenceBench, our next benchmark after PostTrainBench that measures progress on AI R&D automation. AI R&D automation will very likely unfold gradually, starting from “boring” tasks like inference speed optimization that are very easily verifiable (accuracy +
https://x.com/maksym_andr/status/2057106398228439148
A new set of open-weight models is topping the leaderboard for document understanding 🔥 INF just released two models: Infinity-Parser2-Pro (35B) and Infinity-Parser2-Flash (2B) that top our @huggingface leaderboard for ParseBench. Two key insights: ✅ An expanded synthetic data
https://x.com/jerryjliu0/status/2055405690538070340
Agent Evaluation: A Detailed Guide
https://cameronrwolfe.substack.com/p/agent-evals
Better Experiments with LLM Evals — A funnel, not a fork | Spotify Engineering
https://engineering.atspotify.com/2026/5/better-experiments-with-llm-evals-a-funnel-not-a-fork
China’s Hanyuan-2 debuts as ‘world’s first’ dual-core quantum computer — 200-qubit claims incredible power efficiency, but lacks critical performance benchmarks | Tom’s Hardware
https://www.tomshardware.com/tech-industry/quantum-computing/china-claims-worlds-first-dual-core-quantum-computer
Does this transfer to benchmarks? The downstream evals are much noisier, but the trends are generally in line with what we see in the pretraining case, with the pool starting low and eventually catching up.
https://x.com/tatsu_hashimoto/status/2057489440273322447
Huge, did NOT expect that release. Evals looks very solid, significant jump compared to composer 2! But: it’s 10x more efficient than the competition. Looks really exciting. Need to try it out
https://x.com/kimmonismus/status/2056494027189751842
I just published a detailed guide on evaluating agents. It covers: 1. Agent fundamentals (everything from basic concepts to complex ideas like multi-agent systems). 2. Common evaluation patterns / frameworks observed in practice. 3. Case studies of popular agent benchmarks
https://x.com/cwolferesearch/status/2056399847553409301
I think of benchmarks like the heisenberg uncertainty principle between hardware potential and current software state. Luckily, All benchmarks exist on a spectrum between the two extremes. An opinionated thread:
https://x.com/QuentinAnthon15/status/2056450379932647533
I’m wrapping up a writeup on how to evaluate agents. The overview uses Terminal-Bench and Tau-Bench as the primary case studies, but I’m including the following benchmarks as well: – GAIA and GAIA-2 (https://t.co/D9jPQPYYGU): general assistant benchmarks that require reasoning,
https://x.com/cwolferesearch/status/2055437703823372728
Interesting tidbits from the Cerebras listing: Foundation Capital turned a $37m check into a $4.8b return at close. Benchmark partners personally sunk significant amounts into the company, 50% to 60% of the $225m Series H alone. That $225m stake is now worth $786.1m.
https://x.com/shenlucinda/status/2055033736031592843?s=12
LangSmith Engine feels like the missing CI/CD loop for AI agents , automatically detecting failures, clustering issues, proposing fixes, and generating evals from production traces. Agent engineering is evolving fast.
https://x.com/krishdpi/status/2056102370434798034
🆕Daytona’s Agent-Native Compute: 60ms sandboxes, 50K startups in 75 sec, 850K daily runs, RL/evals, CLI > MCP, & the end of localhost
https://t.co/3sauItT6oc @daytonaio CEO @ivanburazin explains why AI agents need composable computers, how Daytona pivoted from human dev
https://x.com/latentspacepod/status/2057565350187995260
turns out that building evals is super super challenging even now. i thought a lot of it was table stakes but turns out it has only become harder since agents are now more complex than ever! going to start tweeting more about how i design evals, especially to create autonomous
https://x.com/palashshah/status/2055410769387303004
We believe the agent harness in @code to be best in market today. Today, the team shared a behind-the-scenes look at our harness optimization efforts, including our own offline evaluation suite vsc-bench:
https://x.com/pierceboggan/status/2055322165969604966
We’re publishing our first end-to-end benchmarks for Zyphra Inference on @AMD Instinct MI355X. Our inference optimizations strongly outperform the AMD baseline and narrows the gap between MI355X and B200 for serving Kimi K2.6, GLM 5.1, and DeepSeek V3.2 🧵
https://x.com/ZyphraAI/status/2056404622483562623
What the benchmarks don’t show: Composer 2.5 is a better collaborator. The model sends more useful updates as it works, and writes succinct, structured messages that surface the context you actually need.
https://x.com/jonas_nelle/status/2056422317740466192
when evaluating long running agents, all of your evals don’t need to be end to end. i’m working on a proper blog about this, but in our evals for our agents that run for 30-60 minutes, we have two sets of evals. the first is end to end, provide inputs and llm as a judge over
https://x.com/palashshah/status/2056449711767265420
You can now run mini-swe-agent on ProgramBench to reproduce all of our baselines. Excited to kickstart more harness innovation! 🧵
https://x.com/KLieret/status/2057471442066030795
Introducing a revival of PapersWithCode! As @ilyasut said, we’re back to the “”age of research””. Hence, it’s important to share research and build on each other’s work. > find SOTA per domain, not just LLMs > leaderboards > methods > all parsed at scale using AI agents.
https://x.com/NielsRogge/status/2056366395605078252
📖 The second half of LLM evaluation has officially begun. A fantastic deep dive from Zhihu contributor 李磊NLP on why the Agent era is forcing us to rethink almost everything about AI evals. 2026 may ultimately be remembered as the year Agents moved from demos → production.
https://x.com/ZhihuFrontier/status/2056408194801635391
The gap between what you can do on ChatGPT/Codex and Claude/Code/Cowork is closing, as Anthropic & OpenAI converge on a single experience. Google’s experiences are diverging: Studio & Gemini & Antigravity & the other Google AI apps are increasingly different. Which will win?
https://x.com/emollick/status/2056879946316886119
Servers account for 60% of the total cost of owning a 1 GW AI data center. A typical 1 GW AI data center costs about $38B in up-front capital and $0.9B/year to operate. Annualizing the capital expenses over equipment lifespans, that equates to $8.5B/year, with $5B for servers.
https://x.com/EpochAIResearch/status/2055029633322779006
Could an AI company lose control of its own agents? To find out, Anthropic, Google, Meta, and OpenAI let us (1) test their best internal models with CoT access, (2) review non-public info about capabilities, alignment, and control. The result: our first Frontier Risk Report.
https://x.com/METR_Evals/status/2056800023149760666
📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇
https://t.co/Y3iSWh4gi2 @AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We’re now extending it to
https://x.com/StevenDillmann/status/2057144415513420049
meh doesn’t even beat Kimi or GLM
https://x.com/scaling01/status/2056795648742076743
model half-life | AI Focus
https://aifoc.us/model-half-life/
State of AI 2026
https://2026.stateofai.dev/en-US
Leave a Reply