“There has been a lot of criticism of LLM scaling recently, but what exactly does the science behind scaling laws tell us? Power laws. LLM scaling laws are based upon the idea of a power law, which is defined via the function below: y = ax^{p} Here, we have two quantities–x and
https://x.com/cwolferesearch/status/1878929929611448588
“Hacking on ollama this weekend. Everything about it screams minimalism. > Number of key/mouse strokes from install to CLI chat to python API cannot be smaller. > “Cognitive perplexity” is very low. Our brains naturally predict what should be the expected action when using a new
https://x.com/DrJimFan/status/1878134226043994163
“AI SOFTWARE ENGINEERS ARE RAPIDLY GROWING UP! Yes, a normal SWE does a lot more than write code… However, AI is growing up very fast -It’s extremely good at studying code bases and coming up with engineering design docs – It can write test cases and test its code – It’s” / X
https://x.com/bindureddy/status/1879017155423080482
Co-Adapting Human Interfaces and LMs
https://jessylin.com/2024/11/12/co-adapting-human-interfaces/
“Compress once, personalize forever – ComMer’s approach to efficient LLM adaptation. ComMer introduces a framework that compresses and merges user data into compact representations for efficient LLM personalization, reducing computational costs while maintaining performance.”
https://x.com/rohanpaul_ai/status/1879480722119123110
“Giving LLMs the ability to consult both their old and new experiences. IMSM introduces a siamese LLM architecture that prevents catastrophic forgetting by interweaving memories from original and fine-tuned parameters during inference. 🤔 Original Problem: →”
https://x.com/rohanpaul_ai/status/1879814440255996043
“Sqrt > Division RoRA fixes LoRA’s scaling problem by using square root, making larger ranks actually useful. LoRA’s performance drops at rank 32, but RoRA keeps improving. RoRA optimizes LoRA’s scaling factor from α/r to α/√r, ensuring better performance with increasing rank”
https://x.com/rohanpaul_ai/status/1878756888088916147
“ThunderKittens makes it easy to write fast deep learning kernels in CUDA Helps users build fast GPU kernels by simplifying matrix operations with warp-group primitives, async memory handling, and register/shared memory optimizations. Supports PyTorch bindings for LLMs and”
https://x.com/rohanpaul_ai/status/1879487547136786902
“Transformer attention reimagined: Element-wise computation beats dot products at their own game Element-wise attention replaces traditional dot product attention with squared Euclidean distance computation, making transformer models faster and more memory efficient. —–”
https://x.com/rohanpaul_ai/status/1879478944531132434
“Your brain’s next 5 seconds, predicted by AI Transformer predicts brain activity patterns 5 seconds into future using just 21 seconds of fMRI data Achieves 0.997 correlation using modified time-series Transformer architecture —– 🧠 Original Problem: Predicting future”
https://x.com/rohanpaul_ai/status/1880184389218496770
“Tensor Product Attention Is All You Need Proposes Tensor Product Attention (TPA), a mechanism that factorizes Q, K, and V activations using contextual tensor decompositions to achieve 10x or more reduction in inference-time KV cache size relative to standard attention mechanism”
https://x.com/iScienceLuvr/status/1879091791753843064
Silicon Valley turn of fortune: Intel worst year, Broadcom record gain
https://www.cnbc.com/2024/12/31/silicon-valley-turn-of-fortune-intel-worst-year-broadcom-record-gain.htm
“A friend at CES showed me @readai_ It is the best note taking app I have seen. Been using it ever since. And while we are talking about that, have you tried @simpleailab yet? What’s your favorite AI app for your phone? I mean other than ChatGPT, Grok, Claude, Perplexity, or” / X
https://x.com/Scobleizer/status/1878504210725285991
“ive had a few tool/script ideas, but im always annoyed that i know i’ll have to add support for anthropic/openai/local models. i wonder if we can converge on a single local endpoint that all apps hit and the user configures that middleware to point to where they want.” / X
https://x.com/rez0__/status/1879339311784726664
“Agents Overview Great write-up on Agents by Chip. Here are my takeaways:
https://x.com/omarsar0/status/1877081848234520827
“Attention has been the key component for most advances in LLMs, but it can’t scale to long context. Does this mean we need to find an alternative? Presenting Titans: a new architecture with attention and a meta in-context memory that learns how to memorize at test time. Titans
https://x.com/behrouz_ali/status/1878859086227255347
Sky-T1: Train your own O1 preview model within $450
https://novasky-ai.github.io/posts/sky-t1/
“Ray 2 produces AI influencers on demand — complete with wide angle vlogging and rolling shutter 🫣”
https://x.com/bilawalsidhu/status/1879638611270008998
“welcome to the frontier! #Ray2”
https://x.com/gravicle/status/1878680835379142960
[2501.06186v1] LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs
https://arxiv.org/abs/2501.06186v1
Things we learned about LLMs in 2024
https://simonwillison.net/2024/Dec/31/llms-in-2024/
Transformer²: Self-Adaptive LLMs
https://sakana.ai/transformer-squared/
“Hybrid linear-softmax attention working very well at large scale and long-context! As we’ve seen with multiple models now, you only need a couple of (full) attention layers” / X
https://x.com/tri_dao/status/1879439184462762225
Why AI language models choke on too much text – Ars Technica
https://arstechnica.com/ai/2024/12/why-ai-language-models-choke-on-too-much-text/
“Transformer²: Self-adaptive LLMs”
https://x.com/hardmaru/status/1879331049383334187
“Scaling Foundation Models with Lightning Attention This paper introduces MiniMax-01, which integrates Mixture-of-Experts; a model with 32 experts and 456B parameters, and 45.9B are activated for each token. Claims match the performance of state-of-the-art models like GPT-4o and
https://x.com/omarsar0/status/1879572512075587872
[2501.06148v1] From discrete-time policies to continuous-time diffusion samplers: Asymptotic equivalences and faster training
https://arxiv.org/abs/2501.06148v1
“Paper shows very small LLMs can match or beat larger ones through ‘deep thinking’ – evaluating different solution paths. Their 7B model beats o1-preview on complex math by exploring 64 different solutions & picking the best one. Test-time compute paradigm seems really fruitful.
https://x.com/emollick/status/1877951754253238636
“Decentralized Diffusion Models UC Berkeley and Luma AI introduce Decentralized Diffusion Models, a way to train diffusion models on decentralized compute with no communication between nodes. Leveraging the associative property of the marginal flow (due to it being a linear
https://x.com/iScienceLuvr/status/1877653288444665915
“On one hand, this paper shows that adding inference-time compute (like o1 does) improves medical reasoning, which is an important finding suggesting a way to continue to improve AI performance in medicine On the other hand, scientific illustrations are apparently just anime now” / X
https://x.com/emollick/status/1879044572837544335
[2501.08712v1] Self-supervised Transformation Learning for Equivariant Representations
https://arxiv.org/abs/2501.08712v1
[2501.06121v1] kANNolo: Sweet and Smooth Approximate k-Nearest Neighbors Search
https://arxiv.org/abs/2501.06121v1
[2501.07301] The Lessons of Developing Process Reward Models in Mathematical Reasoning
https://arxiv.org/abs/2501.07301
“We’re considering adding parameter serving training (PST) support to the JAX backend in Keras — open for feedback. If that’s a feature you want to use, please chime in and say so. Link below.
https://x.com/fchollet/status/1878880859077714382
[2501.05441] The GAN is dead; long live the GAN! A Modern GAN Baseline
https://arxiv.org/abs/2501.05441
[2501.01028] KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model
https://arxiv.org/abs/2501.01028
2501.04227
https://arxiv.org/pdf/2501.04227
2501.06186
https://arxiv.org/pdf/2501.06186
“VoiceFormer, proposed in this paper, reads lips and text to isolate voices in noisy environments, just like humans do VoiceFormer introduces a transformer-based architecture that can separate speech in noisy environments using text content, lip movements, or both, without
https://x.com/rohanpaul_ai/status/1878771375177117871
WebWalker
https://alibaba-nlp.github.io/WebWalker/
“Data Leakage and deduplication are critical when training LLMs! SemHash is a new, blazingly fast semantic text deduplication library that combines Model2Vec embeddings with ANN-based similarity search through Vicinity, making it possible to deduplicate millions of records in
https://x.com/_philschmid/status/1878743789155516565
“Instead of word-by-word or entire essays, this paper teaches RLHF using natural text segments. A novel approach to improve RLHF by assigning rewards to meaningful text segments instead of individual tokens or entire sequences, making reward learning more effective and
https://x.com/rohanpaul_ai/status/1880173612289913140
“Why use two networks when one does the job better? REINFORCE++ proves it REINFORCE++ cuts LLM alignment training time by 30% while maintaining performance Token-level KL penalties + PPO-clip loss = Better LLM alignment without the overhead Simplifies the REINFORCE algorithm
https://x.com/rohanpaul_ai/status/1880178312515097031
Beating cuBLAS in Single-Precision General Matrix Multiplication
https://salykova.github.io/sgemm-gpu
[2501.08313] MiniMax-01: Scaling Foundation Models with Lightning Attention
https://arxiv.org/abs/2501.08313
“When LLMs compete with better models during training, they learn to generate higher quality responses. PoFT (Preference-Oriented Supervised Fine-Tuning) boosts supervised fine-tuning by making target models compete against aligned LLMs on training data, improving model
https://x.com/rohanpaul_ai/status/1880183960392925628
[2501.08643v1] MonSter: Marry Monodepth to Stereo Unleashes Power
https://arxiv.org/abs/2501.08643v1
GMAE
https://brjathu.github.io/gmae/
[2501.00663v1] Titans: Learning to Memorize at Test Time
https://arxiv.org/abs/2501.00663v1
Autoregressive Pre-training from Videos
https://brjathu.github.io/toto/
“Towards Best Practices for Open Datasets for LLM Training On June 11, 2024, Mozilla and EleutherAI convened 30 scholars and practitioners to create normative principles and technical best practices for creating openly licensed LLM training datasets. The group identified seven
https://x.com/iScienceLuvr/status/1879743492450037889
“LLM4AD introduces a unified Python platform that automates algorithm design using LLMs, making it easier for researchers to develop and compare new algorithms across different domains. —– 🤔 Original Problem: → Researchers struggle with implementing LLM-based algorithm
https://x.com/rohanpaul_ai/status/1879812979799335396
[2501.01956] Metadata Conditioning Accelerates Language Model Pre-training
https://arxiv.org/abs/2501.01956
“LLMs get transparent: New architecture , proposed in this paper, reveals reasoning behind every prediction CB-LLM introduces interpretable neurons in LLMs that provide clear explanations for model decisions while maintaining performance, enabling concept detection and controlled
https://x.com/rohanpaul_ai/status/1879480533966889430
“🚨📢 Excited to announce the ICLR 2025 Workshop on Building Trust in LLMs and LLM Applications! 📢🚨 Submit all your papers, and we’ll see you in Singapore! There will be paper awards, and we have a stacked lineup of speakers and panelists.” / X
https://x.com/micahgoldblum/status/1878834198620119443
“@wordgrammer i think this is interesting to consider beyond the meme. what would a bowed billionaire child do with, say, 10-100k gpus? past a node or two you can run any open model, so at this scale you’re either training or doing some orchestrated inference (search) training is hard” / X
https://x.com/aidan_mclau/status/1877705861608452332
“VoiceFormer, proposed in this paper, reads lips and text to isolate voices in noisy environments, just like humans do VoiceFormer introduces a transformer-based architecture that can separate speech in noisy environments using text content, lip movements, or both, without
https://x.com/rohanpaul_ai/status/1878771375177117871
