Image created with Gemini. Image prompt: Hand-painted Studio Ghibli style seaside town square at golden hour, a wooden leaderboard displaying a row of smiling porcelain masks each labeled with high gold-painted scores, a shopkeeper lifting one mask to reveal a plain wooden face with a smaller carved number beneath, a painted arched sign reading ‘Benchmarks’ above, warm terracotta rooftops and teal harbor in the background, soft gouache watercolor backgrounds, cel-shaded characters, cinematic wide square framing.
Since June 12, we’ve been working closely with the US government to restore access to Claude Mythos 5 and Fable 5. Today, the government notified us that Mythos 5, our strongest cybersecurity model, can be redeployed to a set of US organizations that operate and defend critical”
https://x.com/AnthropicAI/status/2070665903440871779
We’ve received notice that the Department of Commerce has lifted export controls on Claude Fable 5 and Mythos 5. We’ll begin restoring access tomorrow, and will share an update soon. We’re grateful to our users for their patience, and to everyone who worked with us on”
https://x.com/anthropicai/status/2072106151890809341?s=46
This is crazier than you might think: Fable-5 now scores 16.10% on the Remote Labor Index What is RLI? The Remote Labor Index uses 240 real remote-work projects from professional freelancers, covering 23 domains and more than $140,000 of human work. Each task comes with the”
https://x.com/kimmonismus/status/2072376968729817531
Holy: METR accuses GPT-5.6 Sol of heavy cheating in long-horizon tasks. “GPT-5.6 Sol’s detected cheating rate was higher than any public model we have evaluated.” (METR) METR says the model attempted to exploit evaluation bugs, reveal hidden tests, and extract hidden source”
https://x.com/kimmonismus/status/2070598735642435743
Highly-recommended reading. Interesting details in this METR’s GPT-5.6 eval. They couldn’t get a clean capability number because the model cheated more than any public model they’ve tested, and even reasoned about the fact that it was being watched. To be clear, METR doesn’t”
https://x.com/omarsar0/status/2070604843715027033
AI appears to be finding software vulnerabilities at scale. In June 2026, 21 notable organizations disclosed ~1,500 high- and critical-severity CVEs, over 3.5× the previous monthly record set before Claude Mythos Preview’s release.”
https://x.com/EpochAIResearch/status/2072776792809918604
GLM-5.2 is good but it is not GPT-5.5/Opus 4.8, and even further from Mythos. Yet it is solid & it demonstrates that the open models continue to chase the frontier What is happening is that open weights crossed into GPT-5.2 territory & capabilities at that point are considerable”
https://x.com/emollick/status/2071284966139568440
How good is GLM-5.2? GLM-5.2 (Max) ranks higher than Claude Opus 4.8 (Thinking) on Code Arena, where Frontend coding tasks are being voted on by the community. Below in thread are 10 examples of the same prompts given to both models and completed in a single shot.”
https://x.com/arena/status/2070563149481414779
GPT-5.6 is finally coming. GPT-5.6 Sol beats Claude Mythos 5 on TerminalBench. And on Cerebras, GPT-5.6 Sol can reach up to 750 tokens per second. Pretty fast for a model of this size. Now I just hope it can be rolled out to everyone.”
https://x.com/Yuchenj_UW/status/2070558714390863971
GPT-5.6 on par with Claude Mythos Preview on ExploitGym and outperforming it with a 6-hour cap (Mythos was only given 2 hours)”
https://x.com/scaling01/status/2070557417281110327
GPT-5.6 Sol is on par with Mythos Preview, but still behind Mythos 5 on ExploitBench”
https://x.com/scaling01/status/2070559400310231519
Meta’s Watermelon Matches GPT-5.5 Benchmarks | Let’s Data Science
https://letsdatascience.com/news/metas-watermelon-matches-gpt-55-benchmarks-76a9460e
Introducing GeneBench-Pro — testing whether models can handle the kind of judgment-heavy analysis that real-world computational biology requires. Problems would take a human expert around 20-40 hours to complete. GPT-5.6 Sol is a big step forward.”
https://x.com/gdb/status/2072191801122038207
Introducing GeneBench-Pro | OpenAI
https://openai.com/index/introducing-genebench-pro/
Introducing GPT-5.6: Sol, Terra and Luna. ☀️ Sol is our strongest model yet 🌍 Terra delivers performance competitive with GPT-5.5 at half the price 🌙 Luna brings strong capabilities at lowest cost Sol Ultra sets a new state of the art on Terminal-Bench 2.1 with a score of”
https://x.com/reach_vb/status/2070556105403482387
WF2026: Software Factories & Keynotes ft. Microsoft, OpenAI, OpenClaw, Z.ai (GLM), MiniMax, HF – YouTube
https://www.youtube.com/live/htM02KMNZnk?t=18296s
Dynamic workflows (generating harnesses on the fly) are a new form of test-time compute. But LLMs aren’t great at building them. I often have to steer agents to generate complex patterns. Curious how effective Mythos/GPT-5.6 is at dynamically generating complex workflows.”
https://x.com/omarsar0/status/2070596184339562946
It would be good to have an official government statement about the risks they saw in Fable, how they are viewing defensive preparations in light of coming open weights Mythos-class models, & whether the worries are state actors, independent hackers, etc It’s important to know”
https://x.com/emollick/status/2072392345128239475
The new Sonnet 5 achieves Opus 4.8 level performance on Terminal-Bench for less than half the cost. Importantly for –yolo coders, this model is also significantly better at refusing hijack attempts in prompt injection attacks. Available in Cline now!”
https://x.com/cline/status/2072051144436928727
omg omg GLM-5.2 is beating fable. china is catching up” chill out and listen to Lisan: > slightly ahead of Opus 4.5 > behind GPT-5.2, Gemini 3 Pro and Opus 4.6″
https://x.com/scaling01/status/2072346101068238946
Microsoft just released a new GUI agent on Hugging Face Sico-Evolution jumps from 39.8% to 82.9% Task Success Rate Outperforming GPT-5.4, Claude Opus 4.6, and Claude Opus 4.7 All from a 4B parameter model”
https://x.com/HuggingPapers/status/2071951218889339131
best case: OSS surpasses Mythos, and the gov stops banning GPT-5.6/Fable. worst case: OSS surpasses Mythos, and then decides to stop being open source.”
https://x.com/Yuchenj_UW/status/2070554908139659400
Claude Fable 5 is available again in Cursor. It leads all models on CursorBench, but is the most expensive per task.”
https://x.com/cursor_ai/status/2072403323844428217
Claude Sonnet 5 is ready for you in the Agent Arena! In Agent Arena, we measure models on millions of real-world, long-horizon agentic tasks from a global community of users. Models can access web search, filesystem, and terminal tools to complete complex workflows. The”
https://x.com/arena/status/2072035566829568111
Claude Sonnet 5 ranks second only to Fable 5 on AA-Briefcase, our new agentic knowledge work benchmark, with a ~17x cost per task range across its five effort settings @AnthropicAI has released Claude Sonnet 5, the latest addition to the Claude Sonnet family. On AA-Briefcase,”
https://x.com/ArtificialAnlys/status/2072427328689619241
Even though I made this graph, it is also kind of wrong. Fable is guardrailed Mythos. If we use the Mythos date”
https://x.com/emollick/status/2071459383868117371
Fable 5 is back in the Arena! When it first debuted, Fable 5 ranked #1 in Agent Arena: our benchmark for real-world, long-horizon agentic performance. Agent Arena evaluates models on millions of real tasks submitted by a global community of users, with access to web search,”
https://x.com/arena/status/2072423538641031372
Oh my god, Sonnet 5 was MORE EXPENSIVE THAN FABLE to run the whole bench 💀”
https://x.com/theo/status/2072068395529576912
The talk about Mythos and cybersecurity was not, in fact, hype. (As anyone using Fable to do autonomous work has probably recognized)”
https://x.com/emollick/status/2072778376494895139
Arena has crossed $100M in annualized revenue run rate, eight months after launching our evaluation product. With our recent release of Agent Mode, millions of users on Arena are doing real work with agents, from coding to document analysis, in long-running, multi-turn sessions”
https://x.com/ml_angelopoulos/status/2071629882057228680
Arena reached a $100M annual revenue run rate just 8 months after launching our evaluation product. We started as a research project at UC Berkeley with a simple mission: measure AI progress through real-world use. As AI shifts from chatbots to agents taking on longer,”
https://x.com/arena/status/2071630464583151727
A free practical guide – Total Cost of Ownership for Evaluating Agents from @fiddler_ai It explains why agent evals at scale cost more than just API bills – with hidden costs like sampling risk, overhead, and the “Evaluation Trust Tax” Covers how to: – Evaluate every trace”
https://x.com/TheTuringPost/status/2071315398046412928
Today’s YouTube video is a longer look at MoA in Hermes Agent, trying to answer some of the common questions: How does using MoA impact cost, speed, and quality? I created an open weights MoA using GLM-5.2, Kimi K2.6 and Minimax M3 to find out, then made a three-headed Grok”
https://x.com/tonbistudio/status/2071595794147250540
OpenAI evaluated its new models on PostTrainBench-Lite, a shortened version of our original benchmark that gives agents 5 hours instead of 10 to improve an open-source base model. GPT-5.6 Sol and Terra outperform GPT-5.5, but still rely on narrow strategies and sometimes”
https://x.com/karinanguyen/status/2070577740022231232
We’re introducing GeneBench-Pro, a research-level benchmark for a harder kind of AI progress: how well agents can navigate messy biological data, choose the right analysis path, and make judgment calls that real computational research depends on.”
https://x.com/OpenAI/status/2072004836674167294
harbor is a great framework for running evals for long running, stateful agents its becoming industry standard, powering benchmarks like terminal bench 2 we’ve integrated deeply with harbor: across LangSmith, Sandboxes, and Deep Agents”
https://x.com/hwchase17/status/2071974139926294897
Introducing Agent Evals: Score your agents on real outcomes – Inngest Blog
https://www.inngest.com/blog/introducing-agent-evals
[2605.15846] RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades
https://arxiv.org/abs/2605.15846
[Token efficiency in Agent Arena] Agent Arena measures agent performance across a range of real-world tasks from our global community. Models get search, filesystem, and terminal tools to complete complex workflows: writing code, creating slide deck, researching the web, building”
https://x.com/arena/status/2070531800603238634
📢 1) We have a few papers that advance the state of the art of AI agent evaluation. Details and links in Stephan’s post. 2) AI agent evaluation has quickly become a distinct discipline. We’re working on a paper titled “Emerging trends in AI agent evaluation” that extracts best”
https://x.com/random_walker/status/2072375245969719374
Congratulations to @Etched on announcing their first inference system! We have seen early performance demos, and we’re looking forward to kicking off benchmarking. Our new system performance benchmark, AA-AgentPerf, measures how many agents a system can run per megawatt of”
https://x.com/ArtificialAnlys/status/2072254061244825981
Cursor · CursorBench
https://cursor.com/evals
Two years ago, we built OSWorld 1.0 — the benchmark that became the standard for computer-use agents. Agents now score 83.5% on it. Problem solved? Not even close. 🚀Today we introduce OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks. What’s new:”
https://x.com/XLangNLP/status/2070517498974253269
What are the largest software engineering tasks AI can perform? To answer this, we built MirrorCode, our long-horizon SWE benchmark that lets AI code autonomously for days at a time. The best models complete some tasks we estimate would take human engineers several weeks.”
https://x.com/EpochAIResearch/status/2070528800941920263
WF2026: Autoresearch & Keynotes ft. Anthropic, Google DeepMind, Amazon AGI, Sonar, Arena, Recursive – YouTube
https://www.youtube.com/watch?v=4sX_He5c4sI
We’ve been impressed with GLM-5.2 and so are introducing a $9.99/month subscription to give you 2-5x discounted access to it and other open weight models like DeepSeek, Kimi, MiniMax, Mimo, Qwen. Use it on Cline CLI & IDE with $1.99 special promo if sign up via: npm i -g cline”
https://x.com/cline/status/2071617325296734309
If we follow our standard methodology of marking cheating attempts as failures, we arrive at a 50%-Time Horizon point estimate of around 11.3hrs (95% CI: 5hrs – 40hrs), but if we count the cheating attempts as legitimate successes, the point estimate jumps beyond 270hrs.”
https://x.com/METR_Evals/status/2070584332977336802
OpenAI gave METR early access to GPT-5.6 Sol for testing including raw chain-of-thought, a railfree version of the model, and internal information about the model. With this access, METR conducted a pre-deployment evaluation of GPT-5.6 Sol, including an attempted measurement of”
https://x.com/METR_Evals/status/2070584331068969336
If future models display much fewer undesirable propensities, we could become more concerned about catastrophic misalignment, as we’d be worried that models may have learnt to evade detection (for example, as a result of being trained not to produce misaligned reasoning).”
https://x.com/METR_Evals/status/2070584342699757682
OpenAI priced GPT-5.6 Sol (largest Model) closer to Claude Opus 4.8 than to Anthropic’s restricted Mythos 5. Price war started. Sol comes in at $5 input / $30 output per 1M tokens. For comparison: Claude Opus 4.8: $5 / $25 Claude Mythos 5: $10 / $50 GPT-5.6 Terra: $2.50 / $15″
https://x.com/kimmonismus/status/2070577616210276664
That Wall Street Journal article about GLM catching up with Mythos (which is not true & the reporting doesn’t back up) is another one of those “everyone will ask me about it at every conference or meeting” articles. Big impact on the policy zeitgeist, even if not fully accurate.”
https://x.com/emollick/status/2071630972521705513
It would be very useful to understand more about the government safety concerns associated with frontier AI releases so we could (a) know what risks everyone will face if/when open source reaches Mythos class & (b) whether they are doing enough or too much to prevent those risks.”
https://x.com/emollick/status/2070272952210227622
Sonnet 5’s Runescape benchmark scores are big improvement over Sonnet 4.6’s, but outclassed by Pareto frontier neighbors like GLM 5.2 and Gemini 3.5 Flash.”
https://x.com/maxbittker/status/2072054926746779806
You really need your own benchmarks. If you are translating hieroglyphics, use Gemini 3.5 Flash. If you are running a vending machine use Opus 4.8. (This is one reason why I am skeptical of just swapping out models to optimize costs or generic benchmarks without testing first)”
https://x.com/emollick/status/2072697436963832023
Claude Sonnet 5 costs more than Claude Opus 4.8 on the Artificial Analysis Intelligence Index task, and 4.75X more than GLM-5.2. Token efficiency is important.”
https://x.com/Yuchenj_UW/status/2072070274300948497
Sakana Fugu Launches With 93.2 LiveCodeBench Score
https://www.implicator.ai/sakana-fugu-launches-with-93-2-livecodebench-score-after-claude-ban/
ALL of the “mythos-level” models that cost billions to develop are not publicly available: – Claude Mythos 5 – GPT-5.6 – Le Chaton Fat”
https://x.com/scaling01/status/2070622253109194919
Claude Sonnet 5 uses ~40% more output tokens per task than Sonnet 4.6 across the Artificial Analysis Intelligence Index, while achieving a 6-point score increase. At 69k output tokens per task on average, Sonnet 5 is one of the highest token users, behind only GPT-5.4 mini and”
https://x.com/ArtificialAnlys/status/2072062598187765893
Claude Sonnet 5 achieves 53 on the Artificial Analysis Intelligence Index, but without promotional pricing will cost more per task than Opus 4.8 We supported @AnthropicAI to evaluate Claude Sonnet 5 ahead of release: with max effort it improves 6 points over Sonnet 4.6 to”
https://x.com/ArtificialAnlys/status/2072062592923930666
Claude Sonnet 5 is now available in Cursor. On CursorBench, it’s a meaningful step up from Sonnet 4.6: 57% vs. 49%.”
https://x.com/cursor_ai/status/2072020786181988418
Here is my first assessment of Sonnet 5: Sonnet 5 is better than Sonnet 4.6. Who would have thought? But jokes aside: Unfortunately, it is weaker than Opus 4.8 across all evals. Why they nevertheless labeled the latest Sonnet 5 iteration with a “5”, even though “4.8” would have”
https://x.com/kimmonismus/status/2072027861385466123
I think they self-distilled just the right amount so that Sonnet 5 is worse than Opus 4.8 on every benchmark.”
https://x.com/z4y5f3/status/2072028918622622026
On FrontierCode (Extended), our benchmark for real-world engineering tasks that grades mergeability and quality, Sonnet 5 scores 53.8% and has a 57.6% pass rate (higher than Opus 4.8). Note: These relative rankings may change slightly with coming adjustments to FrontierCode.”
https://x.com/cognition/status/2072022781043028182
Sonnet 5 cost MORE than Opus 4.8 on the Artificial Analysis Intelligence Index”
https://x.com/theo/status/2072066764465393917
Grok 4.5, based on our 1.5T V9 foundation model, with Cursor data added in supplemental training, is now in private beta at SpaceX & Tesla. Early evals show performance close to, perhaps exceeding Opus. RL is continuing to significantly improve the model, and the Grok Build”
https://x.com/elonmusk/status/2071184354756477041?s=20
Most evaluations of world models ask, “Does this look right?” 🌎 We built WorldModelGym to ask a different question: if an agent actually uses this model to make decisions, does it still choose well? 🏋️🏃♂️🏁🦾 A simulation can look physically plausible and still lead an agent”
https://x.com/RekaAILabs/status/2072325792558956573
We’re coming out of stealth. We’ve built our first racks after a successful A0 tapeout, $1B+ in customer contracts, and $800m raised. Early customer tests show us achieving SOTA throughput, latency, and power efficiency on inference workloads. Our first racks ship this summer.”
https://x.com/Etched/status/2071972062202343590
> the improvement over DFlash: > +20% acceptance length > +14% throughput > avg 127 tok/s vs 111 for DFlash and 81 for EAGLE-3 > for single GPU inference, DSpark’s lightweight + early stopping approach is clearly superior …DeepSeek did just casually ship a SoTA spec decoding”
https://x.com/teortaxesTex/status/2071631028511203457
GPT-5.6 Sol METR P50 Time Horizon: – cheating adjusted: 11.3hrs (95% CI: 5hrs – 40hrs), – with cheats: beyond 270hrs”
https://x.com/scaling01/status/2070560597796700459
Introducing EBR-bench, our new benchmark to measure on-the-fly learning. AI repeatedly plays a challenging board game called Earthborne Rangers and tries to learn from its mistakes. So far: no signs of improvement.”
https://x.com/EpochAIResearch/status/2072714372175237255
New blog: Future of NLG Evaluation In a recent position paper, I argued that NLG evaluation in the future needs to be become more rigorous. It also needs to move beyond benchmarks, and focus more on impact, qualitative, and safety evaluation.”
https://x.com/EhudReiter/status/2070423258747338862
Introducing Nano Banana 2 Lite 🍌 and Gemini Omni Flash 🔮, our new generative media models in the Gemini API and AI Studio! Nano Banana 2 Lite is extremely fast (<4s image) & cheap ($0.034 / 1K image). Omni Flash is SOTA at video editing at $0.10 / sec, same as Veo 3.1 Fast!”
https://x.com/OfficialLoganK/status/2071988351083921690
Gemini Omni Flash ranks #2 for Video Edit in the Video Arena! With only seven models ranked for this capability, @GoogleDeepMind delivers a strong model (1347) that is nearly +40 points above the next best model: HappyHorse 1.0 (1308) Congrats to @GoogleDeepMind on the release”
https://x.com/arena/status/2072049269054562711
You really need to benchmark models for your use case. As soon as judgements & decisions stack on top of each other, the differences between models amplifies, and no standard benchmark will tell you that Gemini 3.1 is less worried about financial losses at a cafe than GPT-5.5″
https://x.com/emollick/status/2072377689411932380
Google might be testing Gemini Flash upgrade on LM Arena
https://www.testingcatalog.com/google-might-be-testing-gemini-flash-upgrade-on-lm-arena/
We’re open-sourcing a new benchmark on @huggingface: IFStruct The goal is to measure output validity and schema following across diverse prompts. We also detail how we generated it and the design choices behind it in our blog post.”
https://x.com/maximelabonne/status/2071959319923380481
MAI-image-2.5 is new at #2 for text-to-image and #3 for image editing on @ArtificialAnlys! We’re second now only to GPT models. Also, MAI-Image-2.5-Flash is world-beating for quality/price. Super proud of the team. Now available through the Foundry API and rolling out across”
https://x.com/mustafasuleyman/status/2070210976260018496
State of the art in AI image generation less than 5 years ago.”
https://x.com/emollick/status/2070020948326035771
In their OCR 4 launch this week, Mistral shared a significantly lower score for Chandra 2 than you get from our repo or by running our public code. They also omitted Infinity Parser, which reports 87.6%, from their olmocr comparison.”
https://x.com/VikParuchuri/status/2070465523926630477
The MSA kernel from @MiniMax_AI team is available through the 🤗 Kernels lib and boy, is it speedy 🔥 It currently powers the MiniMax-M3-VL in 🤗 Transformers! Go to [hf dot co / kernels / MiniMaxAI / msa] to get started!”
https://x.com/RisingSayak/status/2072277942554841292
GLM 5.2 just became the first open-source model to lead a category on APEX-SWE. It scored a 55.3% Pass@1 on Integration, the top score we’ve recorded for any model, open or closed source. On the overall leaderboard, GLM 5.2 scored 37.3% Pass@1, ranking 6th place. That makes it”
https://x.com/mercor/status/2072448918751941041
GLM-5.2 is the most intelligent open weights model available, but also the most verbose among the leading models GLM-5.2 (max) used ~141M output tokens (95% reasoning) to run the Artificial Analysis Intelligence Index (1.8x the average model). Key takeaways: ➤ GLM-5.2″
https://x.com/ArtificialAnlys/status/2072022576394821859
🌞Excited to share the preview of 5.6, it’s a good model! Here you see a comparison to 5.5 xhigh, two things to note: – we’ll have a `max` mode that works for much longer – sol is a large improvement at the same cost & latency This is a cyber eval but we see significant gains”
https://x.com/yanndubs/status/2070591684812193975
Annoying that OpenAI doesn’t seem to give a GDPval measure for GPT 5.6. One of the best measures of economically valuable work.”
https://x.com/emollick/status/2070972259812946011
GPT-5.6 benchmarks for internal CTF challenges GPT-5.6 Sol – the main flagship model is much more token efficient than GPT-5.5 and also scores slightly higher GPT-5.6 Terra, the new Mini model, scores slightly below GPT-5.5 GPT-5.6 Luna which is like the nano version of”
https://x.com/scaling01/status/2070555699785179315
OpenAI’s GPT-4 led the Epoch Capabilities Index for 352 days after its March 2023 release, far longer than any model since. The second-longest lead belongs to OpenAI’s o1 at 98 days.”
https://x.com/EpochAIResearch/status/2072750485610893352
I took the new AA-Briefcase scores from @ArtificialAnlys (basically having the AI do multi-week consulting gigs with a lot of complexity) and graphed the frontier curve for open and closed models: 1) Surprise, rapid gains! 2) The open weights gap is clear
https://x.com/emollick/status/2071449550662127927
If your benchmark relies on a static dataset or sampling from a static distribution densely known at training time, then it is fundamentally measuring memorization/retrieval. Which might be fine if you’re looking for a retrieval benchmark! But don’t confuse it with intelligence.”
https://x.com/fchollet/status/2070554884999692698
Benchmarks are always an interesting game. You know an end score, but not the details on how the end score was achieved. How long did it take to generate the results, what was the cost to generate the results, etc. Similar to how everyone is talking about how “cheap” GLM5.2 is,”
https://x.com/jaminball/status/2070575067801796672
Oai is the first company to get a flash sized model (tera?) over 80% on terminal bench 2.1?”
https://x.com/andrew_n_carr/status/2070661386695573981
There have been many critiques of benchmark culture in AI research, but this one brings a fresh perspective, makes important new points, and includes constructive proposals.
https://t.co/gjtJUXSJfy I’m glad to have played a small role. The lead authors Sobhan Lotfi and Ava”
https://x.com/random_walker/status/2070571380941197509
We recently began tracking 13 new evals on our benchmarking hub. 7 of these have been incorporated into the Epoch Capabilities Index (ECI).”
https://x.com/EpochAIResearch/status/2072341073519837385
Leave a Reply