Image created with gemini-2.5-flash-image with claude-sonnet-4-5-20250929. Image prompt: A cinematic photograph of a minimalist digital scoreboard displaying illuminated performance metrics and progress bars in deep blue and slate grey, with two elegant lit birthday candles placed on top casting warm red-orange light across the cold numerical displays, shot with dramatic side lighting creating high contrast between the technical precision and celebratory warmth, modern and stylish composition.
A SOTA moment to me: Kimi’s OK Computer generate this website in just one shot > It designed a very beautiful site, all images were AI-generated, and when you click, the sidebar expands. > Inside the sidebar, there’s a handwritten letter, it really feels like a website made by a https://x.com/crystalsssup/status/1971133240619757794
Say hi to OK Computer, Kimi’s agent mode 🤖🎸 Your AI product & engineering team, all in one. ✨ From chat → multi-page websites, mobile first designs, editable slides ✨ From up to 1 million rows of data → interactive dashboards ✨ Agency: self-scopes, surveys & designs ✨ https://x.com/Kimi_Moonshot/status/1971078467560276160
Not to take away from Grok 4 Fast (which seems like a very good model) or from Artificial Analysis (one of the few organizations doing independent benchmarking), but the Intelligence Index is an average of pretty saturated benchmarks (aside from HLE), we really need better ones.”” / X https://x.com/emollick/status/1969270709361733942
I find it unimaginably based that the OAI Evals team keeps making benchmarks finding that Claude is better and publishing it anyway. they are 3 for 3 this year in acknowledging specifically how much Claude is better at tasks OAI care about. there is no sarcasm here folks. this https://x.com/swyx/status/1971404125553242253
GPT-5 is the best model for code quality out there 2 years ago, we created the world’s hardest software design quiz. Only 5 questions, multiple choice. Yet only about 3% of software engineers get them. The average score is somewhere between 2 and 3. Supposedly brilliant models https://x.com/jimmykoppel/status/1968683689421701413
Measuring the performance of our models on real-world tasks | OpenAI https://openai.com/index/gdpval/
💥 Announcing GDPval, a new eval that measures model performance on economically valuable, real-world tasks across 44 occupations.”” / X https://x.com/kevinweil/status/1971250647778635904
GDPval.pdf https://cdn.openai.com/pdf/d5eb7428-c4e9-4a33-bd86-86dd4bcf12ce/GDPval.pdf
Just released GDPval: an early step towards better methods for measuring and forecasting real-world model progress.”” / X https://x.com/gdb/status/1971301844585676930
opus 4.1 beats gpt-5-high on OAI’s own new GDP eval. nice of them to be transparent hehe https://x.com/dejavucoder/status/1971253593404735706
Today we’re introducing GDPval, a new evaluation that measures AI on real-world, economically valuable tasks. Evals ground progress in evidence instead of speculation and help track how AI improves at the kind of work that matters most. https://x.com/OpenAI/status/1971249374077518226
🚨 New Models Update! 🔥 Qwen3 coming in hot into the Arena with three different models: 🔹Qwen3-VL-235b-a22b-thinking for Text & Vision 🔹Qwen3-VL-235b-a22b-instruct for Text & Vision 🔹Qwen3-Max-2025-9-23 for Text Check out the thread to learn more about them and get https://x.com/arena/status/1970920636957831611
VraserX e/acc on X: “GPT-5 just passed what researchers call the “Gödel Test.” That means it’s not just solving textbook problems, it’s tackling open math conjectures that would normally take a skilled PhD student days to crack. In a new paper, GPT-5 was tested on 5 unsolved optimization https://t.co/4lGYKLrdrD” / X
https://x.com/VraserX/status/1970902050931159184
xAI has released Grok 4 Fast – breaking through our intelligence vs cost frontier by achieving Gemini 2.5 Pro level intelligence at a ~25X cheaper cost Intelligence: @xai shared with us pre-release access to Grok 4 Fast. In reasoning mode, the model scores an impressive 60 on https://x.com/artificialanlys/status/1969180023107305846
Introducing our Agentic Leaderboards. These new leaderboards test AI agents in real-world, high-complexity environments, setting a new standard for completing end-to-end digital tasks. https://x.com/scale_AI/status/1969416303015301128
👏We are proud to share that in an internal blind test, where professionals evaluated pairwise comparisons using a win/loss ratio benchmark, the Kling AI 2.5 Turbo model significantly outperformed Seedance 1.0, Veo 3 Fast, and Seedance 1.0 Mini in both text-to-video and https://x.com/Kling_ai/status/1970832920085753893
Factory are one of the most formidable teams in the AI Coding space and I think the Droid concept strikes exactly the right balance between concerns of engineer replacement and the ill defined expectations of agents. while there is some way to go on SWEBench still, its p clear https://x.com/swyx/status/1971310686585356295
Kimi Infra team dropped K2 Vendor Verifier > You can visually see the difference in tool call accuracy across providers on OpenRouter. https://x.com/crystalsssup/status/1971158566343184511
LIMI: Less Is More for Agency • Argues agentic AI doesn’t need more data, just better data • 78 curated demos → 73.5% on AgencyBench (beats models trained on 10k samples) • Outperforms SOTA (Kimi-K2: 24.1%, DeepSeek: 11.9%, Qwen3: 27.5%, GLM-4.5: 45.1%) • Establishes Agency https://x.com/arankomatsuzaki/status/1970328242688246160
China’s Alibaba just dropped an opensource 30B agentic LLM that outperforms Claude 4 Sonnet, DeepSeek v3.1, Kimi k2 on a range of agentic search benchmarks. Only 3B parameters are activated per token. 100% open-source. https://x.com/unwind_ai_/status/1969053988143477186
New, very needed benchmark from @scale_AI: SWE-Bench Pro Includes: – Multi-file edits – 100+ lines changed on average – Complex dependencies across large codebases Current top model scores: – GPT-5: 23.3% – Claude Opus 4.1: 22.7% – Others drop further (<15%)”” / X https://x.com/alexandr_wang/status/1969805196462358919
Results so far No single model dominates: GPT-5 “high” reasoning leads on tough tasks but collapses on time-critical ones. Claude-4 Sonnet balances speed vs accuracy but at higher cost. Open-source models (like Kimi-K2) show promise in adaptability. Scaling curves plateau, https://x.com/omarsar0/status/1970147904087322661
@OpenAI Interesting: 1. Linear progress across OpenAI generations (GPT-4o, o3, GPT-5) 2. Claude Opus 4.1 is on top, nearing industry expert, much better than GPT-5 high. Thanks for acknowledging competitors. https://x.com/Yuchenj_UW/status/1971254164069212231
Language Models that Think and Chat Better Proposes a simple RL recipe to improve small open models (eg, 8B) that rivals GPT-4o and Claude 3.7 Sonnet (thinking). Pay attention to this one, AI devs! Here are my notes: https://x.com/omarsar0/status/1971215698140819516
Price analysis reveals trends in the Speech to Text market: Fireworks and Groq are the lowest-cost inference providers for Whisper Large v3, offering competitive access to OpenAI’s model. As word error rate decreases, pricing tends to increase, reflecting the performance-cost https://x.com/ArtificialAnlys/status/1971232403973943517
Clementine just dropped an incredibly useful guide to evaluations in 2025 ✨ The key insight: we’re transitioning from testing knowledge retention to measuring practical problem-solving ability. Her framework spans core capabilities, integrated assistant tasks, adaptive”” / X https://x.com/joelniklaus/status/1968596729852231813
Happy to start to see new, harder AI benchmarks coming into use. But they all need the following things: 1) Validated comparison: how does a human expert do? 2) Public & private tests 3) Care to eliminate question/answer errors 4) Reporting standard errors 5) Real-world validity https://x.com/emollick/status/1969938204879896622
I have been wondering if there is an underlying capability factor that all the many benchmarks for AI are measuring. It seems like the answer is yes. Overall correlation is good (median r ≈ 0.51) and there are distinct clusters (eg reasoning, code) with VERY high correlation. https://x.com/emollick/status/1969986020042010640
Introducing SEAL Showdown | Scale https://scale.com/blog/showdown
Scale AI launches Seal Showdown, an alternative to LMArena leaderboard | Mashable https://mashable.com/article/scale-ai-seal-showdown-benchmarking-leaderboard-lmarena
SWE-Bench Pro https://static.scale.com/uploads/654197dc94d34f66c0f5184e/SWEAP_Eval_Scale%20%289%29.pdf
Last week we found an issue with SWE-Bench, allowing agents to cheat by looking at future commits. Instead of celebrating the SWE-Bench Devs for quickly fixing the issue and being transparent, the HN crowd is dunking on them and drawing wildly inaccurate conclusions about”” / X https://x.com/TacoCohen/status/1966421688846778561
What are popular AI coding benchmarks actually measuring? – nilenso blog https://blog.nilenso.com/blog/2025/09/25/swe-benchmarks/
We’re launching the Artificial Analysis Word Error Rate Index (AA-WER), our new synthesis benchmark for Speech to Text model accuracy comprising of 3 challenging datasets AA-WER comprises three challenging datasets aligned with real-world use cases: AMI-SDM (multi-speaker https://x.com/ArtificialAnlys/status/1971232397921534141
EngDesign: Benchmarking LLMs toward Engineering AGI • 101 tasks across 9 domains (OS, circuits, robotics, etc.) • Simulation-based eval (SPICE, FEA, etc.), not just Q&A • Iterative refinement boosts pass rate (o3 → ~60%) • Error analysis: 111 failure modes identified https://x.com/arankomatsuzaki/status/1970326076271513805
In Defense of AI Evals, for Everyone https://www.sh-reya.com/blog/in-defense-ai-evals/
OSF | Quantifying Human-AI Synergy https://osf.io/preprints/psyarxiv/vbkmt_v1
How are developers using AI? Inside Google’s 2025 DORA report https://blog.google/technology/developers/dora-report-2025/
EmbeddingGemma: lightweight SOTA embeddings • 308M params, built on Gemma 3 • Tops MTEB (multilingual, English, code) <500M models • Matches models 2× its size, efficient even w/ 4-bit or 128-dim embeddings • Encoder-decoder init + geometric distillation • Spread-out https://x.com/arankomatsuzaki/status/1971041110446465251
Microsoft presents Measuring LLM inference energy (production scale) • Median cost: 0.34 Wh/query (chatbot) • Long reasoning: 4.3 Wh/query (~13× higher) • Fleet scale: ~0.9 GWh/day @1B queries → ~web search level • Public est. often 4–20× too high • Efficiency gains https://x.com/arankomatsuzaki/status/1971059016878240241
Agent run times aren’t everything. I gave the same high level task to Sonnet 4 and GPT-5-Codex. Sonnet completed the task across 7 files in ~6 minutes. Codex is so far at 8 files but 32 minutes in still…. The Sonnet output was perfectly acceptable.”” / X https://x.com/zachtratar/status/1970625784500130065
Aviro (@aviro_ai) makes enterprise AI agents continuously upskill to deliver on complex tasks. Their runtime layer, Cortex, helped their in-house deep research agent beat OpenAI’s by 70% on enterprise search and top Microsoft’s Deep Research benchmark. https://x.com/ycombinator/status/1968691488222503194
The Illusion of Readiness (Health AI) • GPT-5 & peers ace med benchmarks—but stress tests reveal fragility • Guess answers w/o images, flip under trivial prompt tweaks • Fabricate “reasoning” that sounds right but isn’t • Leaderboard wins ≠ real-world readiness https://x.com/arankomatsuzaki/status/1970684893966516477
Try the new Qwen models in the Arena!”” / X https://x.com/Alibaba_Qwen/status/1971097727477088717
🚨 New Models Alert: WebDev 💻 GPT-5-Codex and Qwen3-Coder-Plus are both now available on WebDev Arena! In the WebDev Arena, you can test out all the best frontier AI coding models on web development tasks. Vote for your preferred response and see how they stack up on the https://x.com/arena/status/1970962780225507775
SWE-Bench Pro A new coding benchmark to replace SWE-Bench verified was introduced 1-2 days ago: https://x.com/scaling01/status/1969792786594509190
The cost of intelligence continues to fall rapidly after new frontiers are reached: Grok 4 Fast brings the cost of the Intelligence Index >60 category down to just $0.2/$0.5 per million input/output tokens We track the lowest priced model for different tiers of intelligence to https://x.com/ArtificialAnlys/status/1970251031373390292
Pretty wild to see the demographic split on YouTube based on the niche you’re in. Left (my vfx channel) vs. Right (my ai channel) https://x.com/bilawalsidhu/status/1970514756944699515
