Image created with Flux Pro v1.1 Ultra. Image prompt: Assembly instruction diagram for precision measuring calipers with digital display, detailed parts view, technical engineering style, steel blue and yellow accent colors, graph paper background, “BENCHMARKS” in bold technical font, measurement scales visible, calibration instructions indicated
Opus 4 + Claude Code + Claude Max plan = best ROI of any AI coding stack right now”” / X https://x.com/alexalbert__/status/1927410913453203946
Since Claude 4 launch: SWE friend told me he cleared his backlog for the first time ever, another friend shipped a month’s worth of side project work in the past 5 days, and my DMs are full of similar stories. I think it’s undebatable that devs are moving at a different speed”” / X https://x.com/alexalbert__/status/1927803598936887686
On two of the most common tests of creativity (the DAT and the AUT), recent models scored well above the average human in creativity, but not as high as the most creative humans There was lots of variability, but better prompts seem to improve performance https://x.com/emollick/status/1927436892770840883
BOND | BOND https://www.bondcap.com/reports/tai
DeepSeek is aiming for the king: o3 and Gemini 2.5 Pro https://x.com/i/web/status/1928067335014793526
On GPQA Diamond, a set of PhD-level multiple-choice science questions, DeepSeek-R1-0528 scores 76% (±2%), outperforming the previous R1’s 72% (±3%). This is generally competitive with other frontier models, but below Gemini 2.5 Pro’s 84% (±3%). https://x.com/EpochAIResearch/status/1928489527204589680
One reason I think AI development is likely to continue at a rapid pace is that there a growing number of research papers outlining promising directions to take if current approaches to improving AI start to run into barriers. This is an interesting one.”” / X https://x.com/emollick/status/1927373420737560848
Claude 4 Sonnet beating o3-preview on ARC-AGI 2 while being <1/400th of the price https://x.com/scaling01/status/1927418304718623180
VideoGameBench Can Vision-Language Models complete popular video games? best performing model, Gemini 2.5 Pro, completes only 0.48% of VideoGameBench and 1.6% of VideoGameBench Lite https://x.com/_akhaliq/status/1927722717068869750
Web Bench – A new way to compare AI Browser Agents https://blog.skyvern.com/web-bench-a-new-way-to-compare-ai-browser-agents/
OFFICIAL BENCHMARKS OUT – we have a new open source frontier approaching O3 and Gemini 2.5 Pro 🔥🔥🔥 https://x.com/i/web/status/1928054949247693219
OPUS 4 NEW SOTA ON ARC-AGI-2 IT’S HAPPENING – I WAS RIGHT Claude 4 models are the first models that effectively use test-time-compute for ARC-AGI-2 https://x.com/scaling01/status/1927818210331521044
Breaking: Claude Opus 4 jumps to #1 in WebDev Arena! A strong comeback from @AnthropicAI – Opus 4 and Sonnet 4 now on top of the chart, surpassing previous Claude 3.7 and matching Gemini 2.5 Pro. Massive congrats to @AnthropicAI🔥 https://x.com/lmarena_ai/status/1927756554188566803
🚨 Sakana + UBC release Darwin Gödel Machine: AI rewrites own code, boosts SWE-bench from 20% → 50%, Polyglot from 14.2% → 30.7%. Open-ended self-editing agents now real. ⚙️ The Details → The Darwin Gödel Machine (DGM) is a self-improving AI framework where agents evolve by https://x.com/rohanpaul_ai/status/1928309781628080346
I built a platform to make it easier for anyone to land their first AI internship — and I used @lovable_dev to do it. Here’s how InternshipsAI was born and how Lovable helped me bring it to life. 🧵 https://x.com/InternshipsAI/status/1921115383257247997
just used codex’s “”Ask”” functionality to understand the usage of a setting across our entire codebase. very easy to underappreciate the value of AI-enhanced code reading compared to that of code writing https://x.com/gdb/status/1926367007772901508
Our friends @crypticcracking analyzed the latest AI models on a freshly released Modern Sudoku puzzle that “contains some really clever initial logic that might be obvious to many human solvers.” Can the latest AIs spot it and use it to solve the puzzle? https://x.com/SakanaAILabs/status/1927358732339425646
Claude 4 underperforming on Aider Polyglot”” / X https://x.com/scaling01/status/1926795250556666341
Introducing InsightfulPMV2, built with @lovable_dev All-in-One CxO Dashboard – Track Revenue, Cost, Sales Forecast & Cash Flow InsightfulPM V2 is a unified, executive-level dashboard built for CxOs Please upvote this at Lovable Launched: https://x.com/PrafulPujar/status/1922279121092042755
Announcing LMEval: An Open Source Framework for Cross-Model Evaluation | Google Open Source Blog https://opensource.googleblog.com/2025/05/announcing-lmeval-an-open-ource-framework-cross-model-evaluation.html
On SWE-bench Verified, a benchmark of real-world software engineering tasks, DeepSeek-R1-0528 scores 33% (±2%), competitive with some other strong models but well short of Claude 4. Performance can vary with scaffold; we use a standard scaffold based on SWE-agent. https://x.com/EpochAIResearch/status/1928489533886058934
The ability of o3 to agentically use tools in sequence in its chain-of-thought in the chat interface remains a huge differentiator among AIs. I am not sure o3 is “”smarter”” than Gemini 2.5 or Claude 4 (both do better websites than o3, for example), but you can see how tool use https://x.com/emollick/status/1925705390731165754
Anthropic has released its Claude 4 family of models: Claude Opus 4 and Claude Sonnet 4. We evaluated both models on a suite of benchmarks. The main highlight is a significant improvement in coding performance for Sonnet 4. Results in thread! https://x.com/EpochAIResearch/status/1927813645343305902
Claude 4 Sonnet might be the VERY FIRST model to significantly benefit from test-time-compute on ARC-AGI 2″” / X https://x.com/scaling01/status/1927425665055302023
It’s happening 👀 Anthropic just announced Claude 4 Opus and Claude 4 Sonnet just now at the Code with Claude event Early testers said it could code autonomously for up to seven hours https://x.com/rowancheung/status/1925591664548356555
🚀 We’re open sourcing Chatterbox – our state-of-the-art Voice Cloning model that includes text-to-speech and voice conversion! In recent testing, 63.75% of listeners preferred Chatterbox over ElevenLabs. Not only is it free and open source (MIT license), it’s demonstrably https://www.resemble.ai/chatterbox/
“Sudoku-Bench,” which was announced in March, is a benchmark to encourage AI to acquire more creative reasoning abilities through a multi-step process. https://x.com/i/web/status/1926905826465161629
ARC Prize – Leaderboard https://arcprize.org/leaderboard
Explore all the benchmarks—both Epoch-evaluated and external—in one place, our AI Benchmarking Hub: https://x.com/EpochAIResearch/status/1928498593725399123
Individual results across our independent intelligence evaluations: https://x.com/i/web/status/1928074992065073567
It’s hard to make scheming evals look realistic for LLMs – LessWrong 2.0 viewer https://www.greaterwrong.com/posts/TBk2dbWkg2F7dB3jb/it-s-really-hard-to-make-scheming-evals-look-realistic
Inference providers aren’t sleeping on the switch. https://x.com/fdaudens/status/1927834963509961041
DeepSeek’s R1 leaps over xAI, Meta and Anthropic to be tied as the world’s #2 AI Lab and the undisputed open-weights leader DeepSeek R1 0528 has jumped from 60 to 68 in the Artificial Analysis Intelligence Index, our index of 7 leading evaluations that we run independently https://x.com/i/web/status/1928071179115581671
Love this approach by @RedHat_AI. We need more trust & validation in AI and this can help! https://x.com/ClementDelangue/status/1928551872027116000
Sigh, it’s a bit of a mess. Let me just give you guys the full nuance in one stream of consciousness since I think we’ll continue to get partial interpretations that confuse everyone. All the little things I post need to always be put together in one place. First, I have long”” / X https://x.com/i/web/status/1928148705145934252
Capgemini and SAP partner with Mistral to deploy AI for sensitive sectors | Reuters https://www.reuters.com/business/capgemini-sap-partner-with-mistral-deploy-ai-sensitive-sectors-2025-05-26/
Welcome to the Image Arena: FLUX.1 Kontext! 🖼️ 🎨 You’ll find that FLUX.1 Kontext Pro can generate AND edit images. Congrats to @bfl_ml on this exciting release. 🌲👏🌲 Check it out in the Arena, and get voting! https://x.com/lmarena_ai/status/1928236637709865217
We’re expanding the Epoch AI Benchmarking Hub with four more external benchmarks: VPCT, Fiction-liveBench, GeoBench, and SimpleBench! These benchmarks test visual physics understanding, Geoguessr ability, long-context comprehension, and reasoning and logic skills. 🧵 https://x.com/EpochAIResearch/status/1928498581305971139
1. Visual Physics Comprehension Test (VPCT) by @ChaseBrowe32432 Can models predict where this ball will fall? VPCT, a set of similar problems with varying ramps and obstacles, shows that models struggle with basic physical intuition that humans find trivial. https://x.com/EpochAIResearch/status/1928498584418398522
Some insights we realized during this project: GQA is now the standard for most models, but you can even cut the KV cache size in half by sharing the K & V, increasing arithmetic intensity. Our GTA leverages decoupled RoPE to retain the quality of GQA at half the KV cache size”” / X https://x.com/tri_dao/status/1928170650838995236
🚀 DeepSeek-R1-0528 is here! 🔹 Improved benchmark performance 🔹 Enhanced front-end capabilities 🔹 Reduced hallucinations 🔹 Supports JSON output & function calling ✅ Try it now: https://x.com/i/web/status/1928061589107900779
DeepSeek dropped DeepSeek R1 v2 this morning! We at Hyperbolic Labs now serve DeepSeek-R1-0528, the first inference provider serving this model on @huggingface. My vibe check: It seems to be the only model that consistently answers “”what is 9.9 – 9.11?”” correctly. 🐋 To whale: https://x.com/Yuchenj_UW/status/1927828675837513793
DeepSeek has maintained its status as amongst AI labs leading in frontier AI intelligence https://x.com/i/web/status/1928071183276159117
DeepSeek has released DeepSeek-R1-0528, an updated version of DeepSeek-R1. How does the new model stack up in benchmarks? We ran our own evaluations on a suite of math, science, and coding benchmarks. Full results in thread! https://x.com/EpochAIResearch/status/1928489524616630483
DeepSeek’s R1 update consolidates the lead of 🇨🇳 Chinese AI Labs in open weights intelligence https://x.com/i/web/status/1928226455424528519
Happy to share 💭 Mixture of Thoughts 💭 A curated, general reasoning dataset that trims down over 1M samples from public datasets to ~350k through an extensive set of ablations 🧑🍳 Models trained on this mix match or exceed the performance of DeepSeek’s distilled models — not https://x.com/_lewtun/status/1927043160275923158
Hey guys! We noticed some of you sharing screenshots and links to our DeepSeek-V3-0526 article on @UnslothAI. The link was hidden and wasn’t meant to be shared publicly or taken as a fact but it seems a few of you were scrapping through the site and uncovered it early! 😅 The”” / X https://x.com/danielhanchen/status/1926966742519091327
Now imagine DeepSeek R2 The 150IQ Tsinghua grads are about to outsmart the 120IQ MIT midwits https://x.com/i/web/status/1928073407943319588
Ollama can now think! 🤔🤔🤔 For thinking models, and especially useful for very thoughtful models like DeepSeek-R1-0528, Ollama can separate the thoughts and the response. Thinking can also be disabled! This is useful for getting a direct response. This works across https://x.com/ollama/status/1928543644090249565
Live in Cline: DeepSeek-R1-0528 It’s showing significant benchmark gains, now matching OpenAI o3 in reasoning tasks (key for Plan mode). We’re excited to observe how these improvements impact real-world coding performance in Cline. https://x.com/i/web/status/1928140455923044636
There will be DeepSeek R1 0528 Qwen 3 8B too matching Qwen 3 235B Thinking in performance too 🤯 Whale COOKED! https://x.com/i/web/status/1928058862923391260
DeepSeek R1 05-28 LiveBench results: – 8th in the Overall ahead of o4-mini, Gemini 2.5 Flash Preview and Qwen3-235B-A22B (biggest competitors) – 1st on Data Analysis !!! – 3rd on Reasoning !! – 4th on Mathematics ! – 11th on Language – 20th on Instruction Following – 23rd on https://x.com/i/web/status/1928173385399308639
Deepseek MLA is afaik the first attn variant that can hit compute-bound regime during inference decode, thanks to high arithmetic intensity (~256). If you’re paying $30k for an H100 and only max out the mem bw and not the FLOPS during inference, you’re leaving $20k on the table.”” / X https://x.com/tri_dao/status/1928170652516725027
We made dynamic 1bit quants for DeepSeek-R1-0528 – 74% smaller 713GB to 185GB. Use the magic incantation -ot “”.ffn_.*_exps.=CPU”” to offload MoE layers to RAM, allowing non MoEs to fit < 24GB VRAM on 16K context! The rest sits in RAM & disk. Quants here: https://x.com/danielhanchen/status/1928278088951157116
My conclusion from the last couple of weeks of AI launches is that none of the AI companies can explain very well what their systems do. This is partially because they don’t always know & partially because there is no established approach for documentation of AI capabilities.”” / X https://x.com/emollick/status/1925715197676691615
introducing: VLM vibe eval 🪭 vision LMs are saturated over benchmarks, so we built vibe eval 💬 > compare different models with refreshed in-the-wild examples in different categories 🤠 > submit your favorite model for eval no numbers — just vibes! https://x.com/mervenoyann/status/1928105481907687891
Multimodal LLMs struggle with complex physics reasoning across intermediate steps and multimodal data. This paper introduces PHYSICSARENA, the first benchmark evaluating models on multimodal physics problems across variable identification, process formulation, and solution https://x.com/rohanpaul_ai/status/1927265658665947443
Can SWE-Agents aid in High-Performance Software development? ⚡️🤔 Introducing GSO: A Challenging Code Optimization Benchmark 🔍 Unlike simple bug fixes, this combines algorithmic reasoning with systems programming 📊 Results: Current agents struggle with <5% success rate! https://x.com/StringChaos/status/1928476388274716707
Following our Sudoku-based reasoning benchmark announcement, we’ve been evaluating the latest models to track improvements in their reasoning capabilities. Today, we’re launching the Sudoku-Bench Leaderboard: https://x.com/SakanaAILabs/status/1926798125060002243
LoRAvatar https://starc52.github.io/publications/2025-05-28-LoRAvatar/
Introducing LisanBench LisanBench is a simple, scalable, and precise benchmark designed to evaluate large language models on knowledge, forward-planning, constraint adherence, memory and attention, and long context reasoning and “”stamina””. “”I see possible futures, all at once. https://x.com/scaling01/status/1928510435164037342
LLMs optimized to fool detectors still exhibit a distinct, detectable writing style. This paper develops a style-aware paraphraser to mimic human styles and evade detection, introducing a metric to measure the human-machine text distribution gap. Methods 🔧: → Train a https://x.com/rohanpaul_ai/status/1926993600480518475
Nemotron-CORTEXA just reached the top of the SWEBench leaderboard for using LLMs to solve software engineering problems, solving 68.2% of SWEBench GitHub issues! It does so by using a multi-step problem localization and repair process, generating multiple proposal candidates https://x.com/ctnzr/status/1927391895879074047
This paper introduces QA-prompting. It first asks the model questions about the article. Then, it uses the article and answers to generate a summary. This highlights key facts, improving accuracy by up to 29% in ROUGE scores. Methods 🔧: → Sample candidate questions and https://x.com/rohanpaul_ai/status/1927350326249578526
SpatialScore https://haoningwu3639.github.io/SpatialScore/
