Image created with gemini-3.1-flash-image-preview with claude-opus-4.7. Image prompt: High-end product photo of a tall glass parfait with perfectly measured horizontal layers of vanilla soft-serve, hot fudge, caramel, and cookie crumble like a leaderboard chart, a tick-marked ruler-spoon with percentage numbers rising from the glass, a crisp placard with bold custom ‘BENCHMARKS’ lettering beside it, soft directional studio light, shallow depth of field, glossy macro detail, classic Dairy Queen red-and-white counter aesthetic, a subtle ’75 — Milford, DE’ embossed on the glass base, landscape composition.
How successfully — and efficiently! — can agents carry out long-horizon tasks on the web? We built a benchmark of ~200 multi-site tasks, based on people’s real browsing history. Many of them take hours to solve. Paper:
https://t.co/yNGw8Fgvbj Led by @JangLawrenceK and
https://x.com/dan_fried/status/2049530695739932876
ParseBench: A benchmark for document parsing agents @llama_index just shipped a benchmark with 2k verified pages for real enterprise documents. Benchmarks are the major underrated component in the ML ecosystem, so I’m excited to see more entities doing open work in the space
https://x.com/osanseviero/status/2048777802015535189
@teortaxesTex @zephyr_z9 FYI:
https://t.co/j5ICoH2Ffe Part 1 Model Analysis Part 2 Analysis of the Network Optimization Scheme Based on 950PR/DT and A3 Clusters Ascend 950DT Benchmarks DS-V4-Flash 284B DS-V4-Pro 1.6T Atlas-A3 Benchmark DS-V4-Flash 284B Part 3 Future Plan
https://x.com/ogawa_tter/status/2047631993702363509
🚀 Better prompts. Smarter agents. More control. The latest @code release introduces the Chat Customizations Evaluation extension, designed to help analyze and refine your prompts, agents, instructions, and skills.
https://x.com/code/status/2049556204930695278
AI evals are becoming the new compute bottleneck
https://huggingface.co/blog/evaleval/eval-costs-bottleneck
Forward and backward benchmark results across common configurations.
https://x.com/Alibaba_Qwen/status/2049462776247247310
I’m not sure what explains Pro’s relative underperformance but people are underrating the Flash. I think even benchmarks are underweighting it. We don’t really have a lot of benchmarks for “”legit 1M context for pennies””. And it can likely be improved far beyond this.
https://x.com/teortaxesTex/status/2047864952862458009
in the day after Whalefall, you’ll see lots of regarded takes from the wypipo and xiaoren. “”x months behind, heh””. “”distillation””. “”benchmax””. “”blackwells””. “”underwhaleming””. All noise. They’ve completed their quest: Solid Ultra-Long Context. Why was it so important? “”Cheap””? no.
https://x.com/teortaxesTex/status/2047623905754448043
KernelBench-Hard coming soon.
https://x.com/elliotarledge/status/2048502965200372132
Let’s talk document formatting. Bold. Italics. Superscripts. Strikethroughs. The visual cues humans rely on every time we read a doc, and ones existing OCR benchmarks completely ignore. 😱””$199″” struck through next to “”$149″” isn’t decoration. It’s the meaning. 😱A superscript
https://x.com/llama_index/status/2049139409316946011
On evals: V4 Flash @ max ~= v4 Pro @ high on reasoning tasks. Pro focuses more on knowledge (simpleQA)
https://x.com/TheZachMueller/status/2047719857869791352
Public free use eval vs Proprietary STD-free eval on onw V4-Pro is “”top three open”” (with a giant gap to 2) on another it’s the only open model even close to the frontier which one looks closer to the truth, anon?
https://x.com/teortaxesTex/status/2047616662879248828
We’re announcing: VibeBench, a new benchmark for what actually matters — how models feel when used on real work by experienced software engineers. But, we need your help. Here’s how it works: 1. An initial cohort of 1000 qualified software engineers (join:
https://x.com/jpschroeder/status/2049139723776495800
You think it’s funny, I think it’s the biggest problem these labs have. They are serving at 20-30 tokens/s and don’t accept more than a couple requests. LisanBench literally takes a day to run.
https://x.com/scaling01/status/2047643015859118167
Don’t try to build a self-improving AI agent without evals. You are just wasting time and compute. An agent can’t improve from traces it can’t evaluate. This is why it’s exciting to see @FutureAGI_ going fully open source with their platform. It combines the best of all the
https://x.com/omarsar0/status/2048759865007591615
Most agentic benchmarks center around tasks that are automatically verifiable. But any task that is veriafiable is also easy to optimize for. This work instead describes the future of critical open world evaluations. Led by @sayashk, our current draft is now live.
https://x.com/sarahookr/status/2048731841759428935
Organizational design for agents is hard, benchmarking agents working in concert is hard. Together, this is the next critical frontier for making AI matter in economically valuable tasks, and we really don’t know very much about it.
https://x.com/emollick/status/2047828327856030047
Every company building on top of AI should be making their own benchmarks. This is the way if you want model progress to disproportionally benefit your company.
https://x.com/OfficialLoganK/status/2048554074107470305
Scores I would like to see from DeepSeek-V4 to confirm it being less than 6 months behind frontier models ARC-AGI-1: ~75% ARC-AGI-2: ~35% GSO: ~26% METR: 4.5-5 hours WeirdML: ~63% basically Opus 4.5 / GPT-5.2 scores
https://x.com/scaling01/status/2047686712051048598
GPT 5.5 (no thinking) scores 67.1% on WeirdML, well ahead of GPT 5.4 (no thinking) at 57.4%, but well behind Opus 4.7 (no thinking) at 76.4%. It’s at the frontier for accuracy/tokens, as it uses less tokens than Opus.
https://x.com/htihle/status/2048717753394090274
GPT-5.5 by @OpenAI is now live in the Arena, landing across multiple leaderboards. Here’s how it ranks by modality: – Code Arena (agentic web dev): #9, a strong +50pt jump over GPT-5.4 – Document Arena (analysis & long-content reasoning): #6, on par with Sonnet 4.6 – Text
https://x.com/arena/status/2048794479646388732
GPT-5.5 is on par with Claude Mythos – GPT-5.5 average pass rate of 71.4% (±8.0%) – Mythos Preview 68.6% (±8.7%) – GPT-5.5 solved a task that takes a human expert ~12 hours in under 11 minutes at a cost of $1.73
https://x.com/scaling01/status/2049870801998864606
GPT-5.5 Pro achieves a new high score of 159 on the Epoch Capabilities Index! ECI is our statistical tool that combines multiple benchmarks into a unified scale.
https://x.com/EpochAIResearch/status/2049186851844771888
GPT-5.5 xHigh is in Battle Mode in the Code Arena. Evaluate models on agentic coding tasks for front-end websites and apps. Scores coming soon!
https://x.com/arena/status/2048846896744247468
Modded-NanoGPT Optimization Benchmark Hundreds of neural network optimizers have been proposed in the literature, recently including dozens citing Muon: MARS, SWAN, REG, ADANA, Newton-Muon, TrasMuon, AdaMuon, HTMuon, COSMOS, Conda, ASGO, SAGE, and Magma, to name a few. The
https://x.com/kellerjordan0/status/2049193527440187494
To clarify, the Arena community evaluated GPT-5.5 with reasoning effort medium (default) and high. The best of GPT-5.5 with xHigh is still incoming! Stay tuned.
https://x.com/arena/status/2048820224938631492
BullshitBench: GPT-5.5 and 5.5-Pro update! They did NOT do well – 5.5 about the same level as GPT-5.4 (around 30-35 rank, 45% pushback). GPT-5.5-Pro did WORSE – only about 35% pushback. I must say the Pro result kind of shocked me. This is actually interesting, what this tells
https://x.com/petergostev/status/2047773402090426548
GPT 5.5 is much smarter than I thought Yesterday, I did one-shots, coding, benchmarks, and was disappointed. Today, I did it all again, except via the API, which is now available. Results changed completely: → one-shot prompts went from bad to very good → excellent coding
https://x.com/VictorTaelin/status/2047818978664268071
GPT-5.5 is now available in Cursor! It’s currently the top model on CursorBench at 72.8%. We’ve partnered with OpenAI to offer it for 50% off through May 2.
https://x.com/cursor_ai/status/2047744579127185843
GPT-5.5 Pro achieves a small bump on GPT-5.4 Pro with 60% lower cost and token use in our frontier science eval, CritPt CritPt tests models on graduate-level physics research problems contributed by 60+ researchers from 30+ institutions globally. When CritPt was released in
https://x.com/ArtificialAnlys/status/2049926072595280030
LisanBench results for GPT-5.5 – it’s good. GPT-5.5 is now the strongest model without Thinking on both metrics! GPT-5.5-medium uses on average ~45.6% less tokens than GPT-5.4-medium while scoring 1.77x higher! (1.14x higher score on the difficulty weighted metric) Running
https://x.com/scaling01/status/2047818395970904229
Opus 4.7 and GPT-5.5 scores on GSO are live! #1 Opus 4.7 @ 42.2% #2 Opus 4.6 @ 37.3% #3 GPT-5.5 @ 37.3%
https://x.com/scaling01/status/2048853227211251891
Summarize 📝0.14.0 is out. GPT-5.5 Fast mode via `–fast`, Reddit thread extraction in the browser extension, local PDF `–extract`, and fixes for auto model config + Meta site compatibility.
https://x.com/steipete/status/2048275589224628677
The new GPT-5.5 is #1 on Terminal-Bench at 82.7. This beats Anthropic’s Mythos Preview scoring 82.0, which they have not released to the public due to cybersecurity and safety concerns. Available in Cline now!
https://x.com/cline/status/2047769312514257148
ARC-AGI-3 testing is done for gpt-5.5 and opus 4.7 Now we’re in analysis mode going through the logs It’s pretty clear where the failure modes are for each model
https://x.com/GregKamradt/status/2049121093307547654
In Expert Arena, GPT-5.5-High ranks #5 – trailing only Claude Opus 4.6 and 4.7. Expert Arena evaluates models on advanced expert-level prompts in the Text Arena, with a focus on real-world professional use cases. This demonstrates GPT-5.5’s strong performance on complex,
https://x.com/arena/status/2048808366810800259
A few more notes on DeepSeek-V4: – it seems to be a ~GPT-5.2/Opus 4.5+ tier model, so they are still ~4-5 months behind the frontier, but ahead of other chinese labs, with Kimi K2.6 being closest – at 1.6T params they now have a model that’s in the same weight class as GPT-5.4
https://x.com/scaling01/status/2047618271310926151
DeepSeek-V4 is definitely better than GLM-5.1 but not quite Opus 4.7, GPT-5.4 or Gemini 3.1 Pro level unfortunately this video had no comparison to Kimi-K2.6
https://x.com/scaling01/status/2047733998714052819
Mistral Medium 3.5 is interesting less for the benchmarks and more for the positioning. Look at who they’re comparing against: Kimi, Qwen, GLM, Claude (Sonnet). Not GPT, not Gemini. And i dont mean that in a negative way! With Aleph Alpha being acquired by Cohere last week,
https://x.com/kimmonismus/status/2049545016784413005
New open-weight model dropped: Ling-2.6-flash by @ant_oss • AIME 2026: 73.85 • HMMT Feb 2026: 49.29 • SWE-bench Verified: 61.2 ~107B MoE, MIT license, on the @huggingface Hub now. Open models hitting 60%+ on SWE-bench Verified is wild. A year ago that was closed-frontier
https://x.com/nathanhabib1011/status/2049466639171690820
[2604.22119] Emergent Strategic Reasoning Risks in AI: A Taxonomy-Driven Evaluation Framework
https://arxiv.org/abs/2604.22119
MathNet – a new interesting global multimodal benchmark from @MIT for mathematical reasoning and retrieval It’s a dataset of 30,676 Olympiad-level problems from 47 countries, 17 languages, and 143 competitions over 4 decades, with expert solutions. It defines 3 tasks: – problem
https://x.com/TheTuringPost/status/2049155956135841862
Leave a Reply