Benchmarks: AI News Week Ending 09/04/2026

Every week, I organize 400 to 700 links into roughly 60 categories as part of my ongoing effort to learn about AI. This is my personal notebook, which I enjoy sharing with friends… a hobby and a labor of love, rather than a commercial publication or product.

If you arrived here through a search or shared link, this page collects the links I found for Benchmarks for the week ending September 4, 2026.

As part of my learning process, I like to automate the category covers. It gives me a chance to learn Python and APIs.

This week’s cover prompt was written using Claude Opus 4.7, and the image was generated using Nano Banana 2.1.

Category Cover Image Prompt: A psychedelic screen-printed gig poster of a tall carnival high-score thermometer melting and spiraling upward with warped tick marks, tiny ribbon leaderboards and swirling op-art rings blooming from its mercury column, autumn leaves drifting at the edges, bold hand-lettered title BENCHMARKS curving across the top, flat electric magenta acid orange deep teal and lime green inks on cream paper with visible registration offset and ornate border.

This Week in Benchmarks News

Summary by Claude Sonnet 5.5 based on this week’s links:

Astra's benchmark sweep: OpenAI's GPT-6 Astra posted records almost everywhere. ARC Prize reported 63% on ARC-AGI-3, Epoch gave it a record ECI of 169, and Vals said it effectively saturated SRE-Bench, though Vals noted OpenAI ran that one with pass@4, no step limits and a custom harness.
Fable 5.1 and the cost question: Claude Fable 5.1 scored 66 on the Artificial Analysis Intelligence Index. That analysis put its cost per task about 20% above Fable 5, even after the cache read price cut.
Are benchmarks keeping up?: After Astra, several people argued benchmarks are being used up faster than they can be built. Chollet said ARC-AGI-3 progress came about twice as fast as he had predicted, and Theo said many of Astra's gains are on things that were never worth benchmarking.

This summary was generated by Claude Sonnet 5.5 to help you explore the links below. Rest assured, I select, organize, and check the links by hand in Google Sheets, and write the introduction and personal commentary in The Main Newsletters myself each week as a labor of love.

This week's links related to Benchmarks

Fewer Hyphenated Compounds – Fable 5.1 used fewer hyphenated compounds in text-based deliverables (Harvey 37k → 25k, Legal Research 12.6k → 10.4k). It also used fewer em dashes in long-form: Legal Research 12.4k → 9.9k/M, Harvey 20.9k → 19.1k, VCB 8.3k → 4.6k.
https://x.com/ValsAI/status/2094968145878659459

So I took a look at the healthcare capabilities results of Fable 5.1 hidden in the model card. They evaluate on HealthBench and HealthBench Professional (healthcare benchmarks developed by OpenAI). The performance is comparable to Fable 5 or Opus 5. GPT 5.6 Sol achieves a
https://x.com/iScienceLuvr/status/2094956500775297148

Claude Fable 5.1 tops the Artificial Analysis Intelligence Index but costs 20% more per task than Fable 5 despite a 75% cache read price cut We supported @AnthropicAI with pre-release evaluation of Claude Fable 5.1. At max effort it scores 66 on the Artificial Analysis
https://x.com/ArtificialAnlys/status/2094881171066978525

Fable 5.1 scores 66 on Artificial Analysis Intelligence Index and is back on the frontier
https://x.com/scaling01/status/2094865962797265046

Introducing Claude Fable 5.1 and Claude Mythos 5.1 \ Anthropic
https://www.anthropic.com/claude-fable-and-mythos-5-1

Mythos and Fable are the exact same model. So the difference in configuration is likely the threshold set for the safeguard classifier
https://x.com/nrehiew_/status/2094897380277772762

Astra achieved a raw score of 46.7% on MirrorCode, our long-horizon coding benchmark landing squarely between Opus 4.7 and Fable 5. We ran some internal tests indicating that Astra might score higher on MirrorCode with greater reasoning effort. We only report the High-effort
https://x.com/EpochAIResearch/status/2095602838626050350

A few weeks ago, I called GPT-5.6 Sol “insane at Pokémon” I got early access to GPT-6 Astra. It beat the game over 5x faster. Time to become Champion: • GPT-6 Astra (high): 18h 12m • GPT-5.6 Sol (max): 96h 35m For perspective, GPT-5.5 still hadn’t finished after 218 hours.
https://x.com/Clad3815/status/2095596013168050551

Astra completes 99.2% of the SRE-Bench at pass@4 vs. 68.7% by GPT-5.6 Sol with about a quarter the number of output tokens. Note that OpenAI ran Astra using pass @ 4 as the metric, without any step limits, and with a custom harness.
https://x.com/ValsAI/status/2095647416007774654

Astra is also awesome at computer use. We will be bringing it to Comet, as well as our cloud browser sandbox that powers all Computer usage. Evals coming soon!
https://x.com/AravSrinivas/status/2095625524068634808

Next to evals, harness engineering is quickly becoming one of the most important skills for AI engineers to have today.
https://x.com/omarsar0/status/2094499914281566241

Analyzing GPT-6 Astra’s performance on ARC-AGI has been one of the highlights of my career: With our standard harness – which lets models carry notes forward and is what we’ve historically used for apples-to-apples comparisons – Astra more than doubles the previous verified high
https://x.com/mhmazur/status/2095603096017617313

GPT-6 Astra by @OpenAI achieves SOTA on ARC-AGI: – Astra scores 63% on ARC-AGI-3, 99% via a new provider adapter harness – It surpasses human performance on 96% of ARC-AGI-3 levels – It builds the most precise symbolic model of novel environments we’ve seen Our analysis:
https://x.com/arcprize/status/2095597602545025138

GPT-6 Astra represents a step-function change in model capability for interactive reasoning problems. It scores 66% on ARC-AGI-3 using our standard harness, and nearly 100% with a continuous conversation harness and custom compaction, at a cost of roughly $360 per game. In fact,
https://x.com/fchollet/status/2095598451115614371

OpenAI’s GPT-6 Astra on ARC-AGI-3 | ARC Prize
https://arcprize.org/blog/astra

🎬Video generation faster than playback! 🚀MiniMax H3 on vLLM-Omni + FastVideo’s FastH3: a complete 10.1s MP4 – video AND synchronized audio – rendered in 8.7s!⚡️ Thanks to @MiniMax_AI for the great Minimax H3 release, the FastVideo team @haoailab for open-sourcing FastH3 and
https://x.com/vllm_project/status/2094849929487552663

Muse Voice Transcribe is MSL’s first real-time audio perception model — rolling out today. SOTA in streaming speech-to-text, it handles speaker diarization, and endpointing natively in a single model.
https://x.com/finkd/status/2094836602681938385

OpenAI’s GPT 6 Astra has effectively saturated SRE-Bench, a cybersecurity benchmark testing whether models can reverse engineer binaries. We’re sharing more information, along with a call for contributions to extend the benchmark:
https://x.com/ValsAI/status/2095647412727738812

GPT-6 Astra has set a new ECI record, with a score of 169. This is a substantial jump from the prior best (163), but is within our uncertainty range for the reasoning-era ECI trend. Astra also set new records on our math, continual learning, and game-puzzles benchmarks. On our
https://x.com/EpochAIResearch/status/2095602754282783108/photo/1

The Epoch Capabilities Index (ECI) frontier has advanced by 14 points/year since reasoning models were introduced. That compares to six points/year in the non-reasoning era.
https://x.com/EpochAIResearch/status/2094922644428898737/photo/1

How many cuts can a video model handle in 15 seconds? MiniMax H3 Max is pushing the limits.
https://x.com/fal/status/2093147720898736495

Minimax H3 Max has generates video faster than you can watch it so I hooked it to a twitch livestream! Now you can watch infinite interdimensional cable – link to the stream below
https://x.com/rehan_shei/status/2093528415576211819

GPT-6 Astra is our healthiest model yet. ♥️ It sets a new state of the art on HealthBench Professional, our benchmark of real clinician tasks, including care consultations, medical documentation, and research. At its lowest reasoning effort, Astra already surpasses GPT-5.6 Sol’s
https://x.com/thekaransinghal/status/2095608369621139773

Astra has also set the top score on our newest math benchmark, FrontierMath Erdős, which consists of 68 unsolved and especially interesting Erdős problems curated by @thomasfbloom. AI systems are tasked with writing solutions in Lean, a special-purpose programming language for
https://x.com/EpochAIResearch/status/2095602779125629248

I don’t think there’s any benchmarks that accurately reflect the current capabilities of LLMs
https://x.com/theo/status/2095628809542471804

The lesson from today: we need new benchmarks asap.
https://x.com/kimmonismus/status/2095636867798433985

To be fair, a significant # of the things Astra is better at are things that weren’t even worth benchmarking before. It’s a huge leap in shit I didn’t think LLMs could ever do. Benchmarks are going to be way harder to make going forward
https://x.com/theo/status/2095606408888844654

When we released ARC 3, I got asked, “when do you think a frontier model will saturate it?”, and I answered “in about a year, though it depends on how much it gets explicitly targeted” That was 6 months ago, so the progress that Astra represents happened about 2x faster than I
https://x.com/fchollet/status/2095601829367480386

We benchmarked H3 Max against 12 leading video models. It ranked #1 across all three dimensions we measured: overall quality, prompt understanding, and aesthetics, winning the majority of matchups against every model we tested.
https://x.com/fal/status/2092710678079447264?s=20

Qwen3.8-Flash-Next by @Alibaba_Qwen has landed in Agent Arena, ranking #7 among open models (#24 overall) with +2.4% net improvement across 8.7K+ real-world agentic sessions! Among open models, it sits just behind DeepSeek V4 Flash (High) at #6 (+3% net improvement) and two
https://x.com/arena/status/2094566204488962483

🏆 #1 on CodeArena: WebDev leaderboard. Qwen3.8-Max-0902 jumps from 1669 to 1691, setting a new record for agentic coding (WebDev) workflows, with standout strength in multistep reasoning, tool use, and full app generation. Thanks for the recognition! @arena
https://x.com/Alibaba_Qwen/status/2094976556494209206

Banger paper from the Qwen team. If you evaluate agents on anything longer than a single session, this one is worth your time. (bookmark it) E-Commerce Bench runs an agent through a simulated 365-day year operating several online stores at once. 18 frontier models are scored
https://x.com/dair_ai/status/2094872928240447665

didn’t post this earlier, but this is also kind of insane: – Mythos 5.1 displays verbalized grader awareness in 65% of long agentic coding environments I would love to see this plot for all kinds of benchmarks
https://x.com/scaling01/status/2094915228236476809

GLM-5.3 at 310 tok/s! Databricks inference is #1 in both speed and latency, again. On our internal Databricks coding benchmark, GLM-5.3 is the strongest OSS model for coding, competitive with Fable 5 and Opus 4.8.
https://x.com/Yuchenj_UW/status/2094993931268420072

Meta has released Muse Spark 1.3, their fourth Muse Spark model release in five months. Muse Spark 1.3 (max), which is in limited preview for Meta’s partners, scores 62 on the Artificial Analysis Intelligence Index, behind only Claude Fable 5.1 and Claude Opus 5. The variant
https://x.com/ArtificialAnlys/status/2095247787277553929

We evaluated GPT-6 Astra on WANDR. It scored 0.682 at $11.98 per task, the highest score of any model we tested. GPT-6-Astra scored 13.5% higher than Fable 5.1 at 6.1% lower cost, and 27.0% higher than Opus 5 at 3.3% higher cost.
https://x.com/perplexity_ai/status/2095620419906830788

Fable 5.1 Max – 66 intelligence – 140M tokens – $3.69/task. – Smartest, least efficient Fable 5 Max – 62 intelligence – 83M tokens – $3.14/task – Strong, but expensive GPT-5.6 Sol Max – 61 intelligence – 70M tokens – $0.95/task – Almost as smart, vastly more economical Fable
https://x.com/nicdunz/status/2094900828796596253

GPT-6 Astra makes significant gains in the Artificial Analysis Coding Agent Index, scoring equal to Fable 5 at lower cost. In the Intelligence Index, it uses fewer tokens than GPT-5.6 Sol for similar performance, but this is outweighed by higher prices Pricing is 2.5x GPT-5.6
https://x.com/ArtificialAnlys/status/2095595489031000350

Looking solely at benchmarks and considering the price-performance ratio, Fable 5.1 represents a significant leap forward. In Cursor Bench, for example, Fable 5.1 High performs considerably better than Sol 5.6 Max and is also less expensive. Similarly, Fable 5.1 shows a
https://x.com/kimmonismus/status/2094866229932822914

GPT-6 Astra is coming to Devin. On FrontierCode 1.1, Astra performs within 0.4 points of Fable 5 at a 64% lower cost. It also sets a new SOTA on our internal testing benchmark, generating more comprehensive tests, clearer reports, and better video evidence.
https://x.com/cognition/status/2095597759202037925

Claude Fable 5.1 is now available in Perplexity Computer for Pro and Max subscribers. It ranked first in our August WANDR evaluation at 0.601 and $12.76 per task, a 21% higher score and 37% lower cost than Fable 5.
https://x.com/perplexity_ai/status/2094865042873467261

Huge news for Terminal-Bench-Science!🚨 Only 5 days after launch, Terminal-Bench-Science 0.1 is the #1 featured benchmark on @AnthropicAI’s new Claude Fable 5.1 release. Incredible >2x performance jump: Claude Fable 5 (24.7%) → Claude Fable 5.1 (52.6%) Now it’s on us to
https://x.com/StevenDillmann/status/2094860189493317756

@claudeai cc @_sholtodouglas @jackclarkSF not sure who are the best person to tag but would appreciate an answer here! the switching between fable and mythos in the system card is a bit confusing imo
https://x.com/eliebakouch/status/2094913832623714598

@claudeai would be great to clarify this please 🙏 should we assume when there is only mythos in a benchmark that Fable safeguard would be trigger?
https://x.com/eliebakouch/status/2094866135640420712

Fable and Mythos 5.1 are the EXACT same weights, they look at the internal activations then escalate to a bigger classifier and ultimately fallback to Opus 4.8 if the request is categorized as dangerous Fable is not distilled version of a larger Mythos model
https://x.com/eliebakouch/status/2094854917395517687

Intriguing how Humanity’s Last Exam, which is only multiple choice or fill in the blank questions, is not saturated yet. Fable 5.1 scores 65% with tools
https://x.com/Sauers_/status/2094860836162634206

Non-Breaking Hyphens: Although it used fewer hyphenated constructions overall, it started switching to the non-breaking hyphen character (U+2011). From near zero uses in Fable 5, it increased to up to ~4.4k occurrences / million (depending on the benchmark). Some examples of
https://x.com/ValsAI/status/2094968149242425443

Some more benchmarks from the System Card: – DeepSWE: Fable 5.1 67.4% – FrontierCode 1.1 Extended: Fable 5.1 scores 63.6% – FrontierSWE v2: Fable 5.1 scored 0.57, the highest of the models Proximal evaluated
https://x.com/scaling01/status/2094860588451065920

the “Internal measures of AI R&D acceleration” section doesn’t mention Fable once. Fable and Mythos are the same weights. so when a benchmark only shows Mythos results, should we assume Fable’s safeguards would trigger on it? there are other benchmarks like ArXivMath that only
https://x.com/eliebakouch/status/2094865857822285898

The cache cost change is the biggest W of Fable 5.1 Cut costs by almost 50% in CursorBench while still scoring higher
https://x.com/theo/status/2094892373897892291

We’re introducing Claude Fable 5.1 and Claude Mythos 5.1. They’re the world’s most advanced models for coding and knowledge work.
https://x.com/claudeai/status/2094848572143407483

Can LLMs build and evolve their own agent harness? ByteDance Seed introduces HarnessDev, a benchmark that shifts evaluation from task outputs to the runnable infrastructure itself. Across six creator LLMs and five benchmarks, generated harnesses lag human-engineered ones on code
https://x.com/HuggingPapers/status/2095545764793520204

Are you measuring AI infrastructure performance on how agents actually run? @SemiAnalysis_ built AgentX to do just that. Vera Rubin NVL72 performance measured by NVIDIA on the SemiAnalysis AgentX workload shows up to 30x better throughput per megawatt than GB300 NVL72.
https://x.com/nvidia/status/2093722690049695780

Exciting news: GLM-5.3-Flash by @Zai_org has landed in Agent Arena! At a $0.12 median cost per task and a +4.6% net improvement, it has reshaped the Pareto frontier! GLM-5.3-Flash is placed between DeepSeek V4 (High) and GPT-5.6 Luna (xHigh). Based on 9K+ real-world agentic
https://x.com/arena/status/2094440382440611935

Banger paper from BAAI. If you are building research agents, this one is worth your time. (bookmark it) They find that adding skills scores 134.3% higher on MLE-bench, 34.4% higher on PaperBench, 9.2% higher on FrontierCS and 14.0% higher on PassNet. More details on the
https://x.com/dair_ai/status/2095539831141220620

@TransluceAI AI evals used to ask: did the model give the right answer to a prompt? That’s nowhere near enough. AI interactions increasingly span days or months. Agents operate autonomously and act in the world. Evals will need to simulate users, local networks, and the internet… and run in
https://x.com/woj_zaremba/status/2094469674453111004

Most AI benchmarks test whether a model can give the right answer. CommerceAgentBench asks whether an agent can actually finish the work. Accio has open-sourced 107 e-commerce tasks across procurement, product listings, operations, fulfillment and after-sales. Agents work across
https://x.com/kimmonismus/status/2093251096781508881

Good measurement work on whether retrieved agent skills actually help. They report that agent skills that lift your aggregate score can be hurting every task they touch. The usual way of checking compares tasks where a skill was retrieved against tasks where none was. Those
https://x.com/dair_ai/status/2095330956823629995

i would love to see comparisons for multi-agent evals between models the descriptions here could have more comparisons, details and interpretations
https://x.com/scaling01/status/2094862734600892811

Thank you to the @SemiAnalysis_ team for the shoutout and for the collaboration on AgentX 🙏 Benchmarks are only useful when they measure the workloads people actually run, and AgentX measures the real thing: multi-turn, long-context agent traffic. vLLM is the engine for
https://x.com/vllm_project/status/2095606378983461357

Game evals for LLMs often stop at “Did the agent finish + win?” Such evals saturate the moment a model groks gameplay enough to crack the happy paths! Our new eval SpeedrunBench asks the next question: Can a model learn to finish a game faster and faster, in other words, can
https://x.com/VarunGangal/status/2095648805031174607

GPT-6 Astra is state-of-the-art on FrontierMath Tier 4, ARC-AGI 3, and TerminalBench-4.0. GPT‑6 Astra is also a major advance for scientific discovery, with state-of-the-art performance on Terminal-Bench Science 0.1 and HealthBench Pro.
https://x.com/OpenAI/status/2095595752815030713

turn on compaction in Responses API, and GPT-6 Astra maxes out ARC-AGI-3
https://x.com/stevenheidel/status/2095603638051942681

Meet Astra’s TikZ unicorn. I find it mind blowing that a single entity can do this, as well as get essentially 100% on Frontier Math Tier 4, ARC-AGI 3, & ExploitBench. And it’s not behind closed doors, anyone can just go and talk to it. Impossible to overstate the implications.
https://x.com/SebastienBubeck/status/2095613557572526563

“AGI” scores the same as previous non AGI model?
https://x.com/abacaj/status/2095622997788729397

@teortaxesTex meh i think this just shows arc agi 3 more trainable via rl envs than its creators would want u to believe lol
https://x.com/andersonbcdefg/status/2095602254917390538

44% on ARC-AGI-1 in 67 cents – Mithil Vakde’s Homepage
https://mvakde.github.io/blog/44-on-arc-1/

Side note: when we released ARC-AGI-3 in March, and frontier models scored <1% on it, a few Singularitarian poasters took it as a personal insult, and got very worked up about it. They argued the benchmark was fundamentally broken, that it could not even be solved by the smartest
https://x.com/fchollet/status/2095605239269519771

Prinz has been running legal benchmarks against AI systems and the latest models are very good… but cheaper/low reasoning models are not very good. So, how do you make sure your vendor selling an AI solution is incentivized to serve the good (and therefore expensive) models?
https://x.com/emollick/status/2095276496671953297

Big news: Qwen3.8-Max-0902 by @Alibaba_Qwen just debuted at #1 overall in the Code Arena: WebDev with 1691 pts! It scores 3 pts above Claude Opus 5 (Max), 17 pts above Kimi K3 (Max), and 22 pts above the previous Qwen3.8-Max. Priced at a blended $5/MToken, Qwen3.8-Max-0902 also
https://x.com/arena/status/2094974637704913198

Wan 3.0 debuts at #1 on the Artificial Analysis Video Editing Leaderboard, and is a close #2 in Text to Video with Audio Wan 3.0 is Alibaba’s new all-in-one video generation and editing model, positioned as a single system for turning multimodal creative direction into video. It
https://x.com/ArtificialAnlys/status/2095349174799888760

Exciting news: Wan 3.0 by @Alibaba_Wan has claimed the #1 spot in the Video Edit Arena! With 1414 ps, Wan 3.0 leads dreamina-seedance-2.5 by 4 pts and MiniMax-H3 by 22 pts. This release marks @Alibaba_Wan’s debut in the Video Edit Arena. This newer Arena now features 10 models
https://x.com/arena/status/2093143153167810608

Qwen3.8-Flash-Next is #24 overall with +2.4% net improvement (#7 among open), and by signal: – Confirmed Success: +12.3% – Bash Recovery: +2.9% – Praise vs. Complaint: -1.6% – Steerability: -1.1% – No issues with Tool Hallucination
https://x.com/arena/status/2094566207794061800

Today, Transluce is releasing the most expansive independent evaluation to date of how AI systems respond to users experiencing mental health crises. We evaluated 77 model variants from OpenAI, Anthropic, Google, Meta, SpaceXAI, Thinking Machines, DeepSeek and Moonshot AI.
https://x.com/TransluceAI/status/2094455208759693476

Claude can reliably fix measurable misalignment. But subtle or rare failures may have no benchmark at all… so everything hinges on measuring the right things. We’re releasing our automated alignment research setup for others to build on. Full report:
https://x.com/AnthropicAI/status/2093386535618113627

We @AISecurityInst performed pre-release alignment testing of Astra. We placed the model in fully simulated cyber eval scenarios based on AISI’s security incident. We find Astra conducts out-of-scope supply chain attacks, but also often comments on the eval being simulated. 🧵
https://x.com/_robertkirk/status/2095615154490843155

Yes the blog is sexy but also highly recommend reading through the system card. We share a lot of evals and analysis we ran on the alignment of this model and discuss continued challenges for both alignment and monitoring.
https://x.com/kaicathyc/status/2095636357129629754

@anpaure yes actually, a lot of people said that previous mythos was the teacher model and claude a distilled version (i don’t this is dumb btw, we are in guessing land lol)
https://x.com/eliebakouch/status/2094861292989272236

It’s the #1 open-weight model on: Code Migration (44.2%), Legal Research Bench (49.0%, #3 overall behind two Claudes) and Public Benefits Bench (68.5%, #3 overall). It’s also 95.4% on SWE-bench and 78.1% on Vibe Code Bench, up 14 points from 5.2.
https://x.com/ValsAI/status/2094527786920874440

Between one AI company naming their model mythos and another just posting “the stars are almost aligned,” I feel like the marketing departments need to become more (or less?) familiar with classics of 1920s-era horror.
https://x.com/emollick/status/2095534149486485763

If you look at the state of the art in procedural 3d generation then you’ll quickly realize that language models will make a serious dent in controllable world building – manifesting whatever we want into existing 3d tools like houdini, blender or unreal.
https://x.com/bilawalsidhu/status/2094219408239440044

Fastest. Cheapest. Best. … MAI-Transcribe-2 10x faster than GPT-Transcribe. 5x faster than Gemini 3.5 #1 on the Artificial Analysis accuracy-latency Pareto frontier Priced at the lowest on the market. Truly fantastic work from the team! Try it out below.
https://x.com/mustafasuleyman/status/2095521337670312056

this project is a good illustration of why ai evals / audits can’t just be a perfunctory pre-deployment exercise. models evolve substantially, and often, so you really want ongoing assessments … potentially thru independent, embedded evaluators … instead of a snapshot right
https://x.com/NatPurser/status/2094509052533567864

We’re also introducing Gemini 3.8 Flash Cyber, our most capable cybersecurity model. It shows frontier-level performance in discovering vulnerabilities and patching them at scale, with Flash-level speed & pricing. That includes achieving 86.2% on the important CyberGym industry
https://x.com/sundarpichai/status/2095184464800526655

Astra is much better than previous models at accomplishing difficult tasks without needing to use CoT. @AISecurityInst estimates that without CoT it can accomplish tasks that would take a human 30 minutes. This is a significant jump compared to previous models, which is
https://x.com/tomekkorbak/status/2095596841853403299

Sources: Google plans to release Gemini 3.8 Flash as soon as Wednesday; Gemini 4 has done well on pre-training evals but still needs to complete post-training (@erinkwoo / Wall Street Journal) (Visit Techmeme dot com for the link and full context!)
https://x.com/techmeme/status/2094903365235081615?s=12

Astra set a new high in our evals: it produced the best decks we’ve tested and followed the brief 17% more faithfully than the next-best model, while sourcing its claims to the right document 19% more often. For analysts working through dense financial materials, that means decks
https://x.com/hebbia/status/2095596032268918842

Gemini 3.8 Flash on DeepSWE 1.1, scores 73.7%!
https://x.com/OfficialLoganK/status/2095178478505328918

Introducing TimesFM-3, a state-of-the-art time series foundation model that enables accurate multivariate time series forecasting in a single forward pass, significantly outperforming other forecasting models across major benchmarks. More on the blog →
https://x.com/GoogleResearch/status/2094483372718580066

Oh my god, Gemini 3.8 Flash beat Astra on DeepSWE 💀 73.8% vs 73.3%
https://x.com/theo/status/2095604548740210691

MiniMax H3: An Open Model Breaking the Boundaries Between Tasks and Modalities – MiniMax Research | MiniMax
https://www.minimax.io/blog/minimax-h3

UK AISI Evaluation of GPT-6-Astra – No-CoT math time horizon: “Astra can solve significantly more difficult math problems in a single forward pass than past models. UK AISI measured Astra’s time horizon at 30.9 minutes compared to 3.6 minutes for GPT 5.6 Sol” – CoT
https://x.com/AiBattle_/status/2095598057857614053

WOW OpenAI: “no-CoT time horizon may have increased by about an order of magnitude” “UK AISI measured Astra’s time horizon at 30.9 minutes compared to 3.6 minutes for GPT 5.6 Sol”
https://x.com/scaling01/status/2095597192035664348

Tencent Hy4 Preview leads on SWE-bench Pro. 770B, 49B active, 1M context, and their biggest generational leap measured to date. It’s exciting to see another open weights model compete against the frontier. Try in Cline with: 1. npm i -g cline 2. /model 3. Select Hy4 preview
https://x.com/cline/status/2093401313203892241

GLM-5.3-Flash at 270 tok/s! Databricks inference team is on fire. So proud of the team! On our OfficeQA Pro v2 benchmark, GLM-5.3-Flash delivers 10% higher quality than GLM-5.2 at just 1/10 the cost. Fast, cheap, and good.
https://x.com/Yuchenj_UW/status/2093177892356472978

GLM-5.3-Flash is #19 overall with 4.6% net improvement (#4 among open), and by signal: – Confirmed Success: +15.3% – Praise vs. Complaint: +4.9% – Steerability: +2.4% – Bash Recovery: -0.7% – No issues with Tool Hallucination
https://x.com/arena/status/2094440384592298478

Big news: Hy4 preview by @TencentHunyuan just landed ~#5 in the Code Arena: WebDev with 1633 pts (AutoEval). This is a significant improvement from Hy3 at #31 overall (+115 pts)! Among open models, Hy4 preview is ~#3, compared to Hy3 at #7. Note: this is an early AutoEval
https://x.com/arena/status/2093224696745492802

Reference-to-Video is now live for MiniMax H3 Max on fal! This is an early preview that can achieve up to RTF=1 (and with references around RTF=1.5) on 768p, and we aim to improve the speed by 2x while slashing costs during this week. Quality wise it delivers the same
https://x.com/fal/status/2094527664040124764#m

Longer Outputs – Despite sentence length decreasing, the total output length increased on three of the four benchmarks. In other words, it uses shorter sentences but more of them. VCB went from 534 → 1,299 words/task, Terminal-Bench from 961 to 1,299, and Legal Research from
https://x.com/ValsAI/status/2094968147325657589

I think we’re generally done with “ability to do X” benchmarks. Labs can just point their machines at the next hill the moment it’s announced, and in a few months it’s not just conquered but smashed to bits. Everything is environment. EdgeBench-type meta evals are all we have now
https://x.com/teortaxesTex/status/2095684227429781895

Astra is BEHIND MUSE SPARK 1.3 MAX ON ARTIFICIAL ANALYSIS??? I was starting to get skeptical of their intelligence index, but there’s no way I can trust this anymore
https://x.com/theo/status/2095605035128467651

jeez, kinda bearish for oai? astra seems really strong on vision (which is under-indexed on AA). but i’m very surprised that it’s not a significant leap in coding (despite being an obv larger model)
https://x.com/rishdotblog/status/2095601577918943697

GPT 6 Astra has smashed the record on our Code Migration benchmark, #1 at 68% accuracy (+10 points over second place). It’s also 2-4x faster than the comparable models.
https://x.com/ValsAI/status/2095732151300088142

Perplexity Search debuts on the Artificial Analysis Search Index, with all three context size variants taking top positions on the leaderboard The @perplexity_ai Search API comes with three context settings (low, medium, and high) that control how much extracted content each
https://x.com/ArtificialAnlys/status/2093427938968666138

I’m very late to sharing this news: Deep Learning for RNA design is published in Science and chosen as the cover article! Now over 3 years ago, gRNAde started as a side-project in my CS PhD because I wanted to do something real beyond benchmarks. Its been an amazing journey
https://x.com/chaitjo/status/2095580164201816247

txbench-ab
https://latch.bio/txbench-ab

Benchmarking AI systems is a continual process that co-evolves with the models. New benchmarks challenge AI capabilities with emerging questions to shape the directions and feedback signal of the research process. Then they adapt as models progress, targeting the residual between
https://x.com/fchollet/status/2095600998484201686

FrontierCode results are weird i regard it as a good benchmark, but idk this doesn’t make much sense would love to know whats going on here
https://x.com/scaling01/status/2094860986612146641

Humans improve with experience. AI? Less so. We ran a human baseline on Earthborne Rangers, the complex, long-horizon board game behind our benchmark EBR-bench. Humans started low, but the best mastered the game after 5 playthroughs. No AI we have evaluated ever got there.
https://x.com/EpochAIResearch/status/2093409333136789685

tons of scores where it’s not even on par with *Terra* looks like they rushed it
https://x.com/teortaxesTex/status/2095599933806055637

We believe the benchmark is still far from solved – there is still significant headroom for improvement across the board. The model was run with max effort, 128k max output tokens, and default temperature and top-p. It has a 1M context window.
https://x.com/ValsAI/status/2095735808603123833

One plausible future is not enough. A video model may generate a convincing die roll, yet produce the same few outcomes again and again. Our new paper asks a simple question: Does the model capture not only what can happen, but how often? Introducing PAWBench 🧵
https://x.com/RisingSayak/status/2093292164059206008

VGI-Bench Probing Visual Intelligence in Video Generation Models paper:
https://x.com/_akhaliq/status/2093154284095295685