Ethan B. Holland

Over 56,600 manually organized AI links and counting

a science fiction pregnancy sonogram. the sonogram shows a humanoid robot fetus. --ar 5:3

Multimodality News: Week Ending 04/12/2024

April 12, 2024

a science fiction pregnancy sonogram. the sonogram shows a humanoid robot fetus. –ar 5:3

“We are building physical AI, a new category of AI that perceives, understands, and reasons about the physical world around us. Curious? Check out video to learn more about Newton, a first-of-its-kind physical AI foundation model. #PhysicalAI #Newton #ArchetypeAI https://twitter.com/PhysicalAI/status/1777468289079464401

“Archetype AI unveiled Newton. It’s a foundation model that fuses multimodal sensor data and natural language to reason about the physical world. In other words, it aims to understand the real world, in real time. https://twitter.com/rowancheung/status/1777880707064299721

“Our vision is to encode the entire physical world using a variety of sensor data, to go beyond text and images to radars, accelerometers, and other sensors. This will allow us to transcend the limitations of human perception and help humanity to make sense of the world around us. https://twitter.com/PhysicalAI/status/1777743330069315720

“Meta announces MA-LMM Memory-Augmented Large Multimodal Model for Long-Term Video Understanding With the success of large language models (LLMs), integrating the vision model into LLMs to build vision-language foundation models has gained much more interest recently. https://twitter.com/_akhaliq/status/1777539936364662817

Stop worrying about LLMs crawling the web. Start worrying about LLMs learning how to use computers and eating the entire concept of user interfaces. There will be no way to “block” AI because AI will be driving the operating systems. Entire new industries and disciplines are coming.

“Apple presents Ferret-UI Grounded Mobile UI Understanding with Multimodal LLMs Recent advancements in multimodal large language models (MLLMs) have been noteworthy, yet, these general-domain MLLMs often fall short in their ability to comprehend and interact effectively with https://twitter.com/_akhaliq/status/1777542957383446691

“🔥 🔥 🔥 Excited to share our latest exploration on metamorphic time-lapse video generation: MagicTime, which learns real-world physics knowledge from time-lapse videos. Github: https://twitter.com/vhjf36495872/status/1777525817087553827

“@mreflow @NathanLands A newly revealed patent from Microsoft Bing detailed ‘Visual Search’. The patent describes a reverse image search with personal results tailored to user preferences and interests. Source: @mspoweruser Link to full patent: https://twitter.com/rowancheung/status/1778271406846824886

“What are the key factors to improve Vision-Language-Models (VLMs)? 👀 @Apple released a very nice paper, “MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training” where they investigated the effects of architecture components and data choices for VLMs. MM1 includes https://twitter.com/_philschmid/status/1776240697219195221

Reverse image on Bing Visual Search may be getting smarter, latest Microsoft patent reveals – MSPoweruser – https://mspoweruser.com/reverse-image-on-bing-visual-search-may-be-getting-smarter-latest-microsoft-patent-reveals/
Geimini 1.5’s audio capability is actually scarily good… : r/OpenAI – https://www.reddit.com/r/OpenAI/comments/1c0a0dv/geimini_15s_audio_capability_is_actually_scarily/

OpenAI

“GPT-4 Turbo with Vision is now generally available in the API. Vision requests can now also use JSON mode and function calling. https://twitter.com/OpenAIDevs/status/1777769463258988634

Gemini

“It can “hear” -> Google’s Gemini 1.5 Pro can now process audio w/out needing a transcript “The model can now listen to uploaded audio files and churn out information from things like earnings calls or audio from videos without the need to refer to a written transcript.” https://twitter.com/glenngabe/status/1777746267667365998

“Can Gemini 1.5 actually read all the Harry Potter books at once? I tried it. All the books have ~1M words (1.6M tokens). Gemini fits about 5.7 books out of 7. I used it to generate a graph of the characters and it CRUSHED it. https://twitter.com/deedydas/status/1778621375592485076

Google’s Gemini 1.5 Pro can now hear – The Verge – https://www.theverge.com/2024/4/9/24124741/google-gemini-pro-imagen-updates-vertex

“I uploaded two music tracks to Gemini 1.5 Pro: Black Sea by Drexciya, and Song of Scheherazade by Renaissance, and asked it to analyse them https://twitter.com/sebkrier/status/1778386521319428167

“🎉 It’s a big day for @Google Gemini. Gemini 1.5 Pro now understands audio, uses unlimited files, acts on your commands, and lets devs build incredible things with JSON mode! It’s all 🆓. Here’s why it’s a big deal 👇 🔈 Gemini can hear Gemini understands audio (up to 9.5 https://twitter.com/liambolling/status/1777758743637483562

Heads up! You’ve scrolled to the end of this category. There may have been just one or two links (above), so go back up and double check to be sure you didn’t quickly scroll down past it.

Be Sure To Read This Week’s Main Post:

This week’s executive overview and top links are here:

AI News #28: Week Ending 04/12/2024 with Executive Summary and Top 48 Links

The post you just read is an deep dive extension of my weekly newsletter, This Week In AI, an executive summary of the top things to know in AI. Each week, I create an accessible overview for laypeople to feel confident they are conversant with the week’s AI developments. I include a curated list of must-click links of the week, to offer everyone a hands-on opportunity to explore the most intriguing updates in artificial intelligence across various categories, including robotics, imagery, video, AR/VR, science, ethics, and more. Beyond the overview, I post these topic-based deeper dives (below). If you haven’t read this week’s overview, I recommend starting there.

Credits/Sources

Most of these weekly links come from just a few prolific oversharing sources. Please follow them, as they work hard to find the news each week and they make it a lot easier for me to compile.