DeepMind Just Changed How AI Sees The World
DeepMind's Gemma 4 eliminates separate encoder networks for vision and audio by projecting raw input patches directly into the main transformer, enabling a tiny local model to handle multimodal reasoning.
Highlights
1 clip · 1 min of 5 min-
- Conventional multimodal AI uses separate dedicated neural networks for vision and audio translation.
- Gemma 4 cuts images into small patches and audio into 40-ms chunks, projecting them directly into the main transformer.
- Removing specialist encoder networks forces the main model to act as eyes, ears, and brain simultaneously.
That’s all the highlights.
Questions this video answers
8 · every answer cites the moment it was said01 Why do massive AI models like DeepSeek fail to see images? 2 sources · 0:06
Giant models often lack native perception capabilities because they rely on text-only architectures. Without integrated sensory components, they cannot process raw visual inputs directly.
02 What is Gemma 4 and how does it differ in size? 2 sources · 0:50
Gemma 4 is an open and free model created by DeepMind that runs locally on laptops. Compared to massive trillion-parameter systems, it is a tiny fraction of the size yet capable of native perception.
03 How do conventional AI models handle multi-modal tasks like vision and audio? 2 sources · 1:48
Conventional setups connect multiple separate neural networks together. They require a dedicated visual model for images and a specialized audio encoder for sound.
04 How does direct patch projection eliminate the need for separate encoders? 2 sources · 2:29
Instead of using dedicated encoders, the model cuts pictures into small patches and projects those pixels directly into its internal representation. Audio is similarly sliced into chunks and poured straight into the main transformer.
05 Why is pouring raw patches directly into the main transformer beneficial? 2 sources · 3:08
This approach forces the system to learn perception and reasoning simultaneously while removing hundreds of millions of specialist parameters. It blurs the boundary between thinking and perceiving, making the AI punch above its weight.
06 What kinds of inputs can Gemma 4 handle efficiently? 2 sources · 1:29
The architecture allows the model to handle both images and audio inputs natively. It can successfully identify objects in videos while engaging in complex conversational tasks.
07 How can smaller open models benefit the broader AI ecosystem? 2 sources · 3:52
Open models serve as valuable gifts to the community and can help massive systems like DeepSeek learn to see more efficiently. They empower scientists, students, and developers to perform their work better.
08 What infrastructure can developers use to run and reproduce AI research papers? 2 sources · 4:47
Developers can use platforms like Lambda to quickly reproduce AI research papers and run experiments. It provides powerful Nvidia GPUs to train models, fine-tune existing ones, or run chatbots efficiently.
Section-by-section summary
3 parts
Massive AI models often lack native perception
Despite having trillions of parameters and costing hundreds of thousands of dollars to run, many large language models cannot inherently see or process raw images without separate pipelines.
Direct patch projection unifies perception and reasoning
DeepMind's Gemma 4 bypasses traditional vision and audio encoders by slicing inputs into raw patches and chunks, feeding them straight into the main transformer's internal representation. This architecture removes hundreds of millions of specialist parameters and significantly boosts efficiency on local hardware.
Open-weight models and GPU infrastructure drive research
Accessible open models empower the scientific community, though their continued availability should not be taken for granted. Developers can leverage specialized cloud GPU platforms like Lambda to quickly reproduce papers and run local experiments.
Connects to your library
by shared topics
n next · p previous clip · t highlights/full
< > slower/faster · / search what was said
-
Gemma 44× · first 0:53
An open and free small local model by DeepMind featuring native multimodal perception.
-
DeepSeek1× · first 0:08
A massive trillion-parameter AI model that initially lacked native visual perception capabilities.
-
Vision transformer1× · first 2:50
A dedicated neural network architecture traditionally used to interpret images.
-
Main transformer1× · first 3:05
The core model architecture that directly receives raw image patches and audio tokens.
-
Lambda2× · first 4:47
A cloud platform providing powerful Nvidia GPUs for running experiments and reproducing research papers.
-
Nvidia GPUs1× · first 5:07
Powerful hardware accelerators used to train models, fine-tune them, and run inference.
790 words · about 3 min to read
Massive AI models often lack native perception
Finally, something amazing happened. You see, today AI models have become almost comically large. Some of the biggest open and free AI models, like DeepSeek, reached over 1.6 trillion parameters in size. And that's not even the biggest one. So, these cost hundreds of thousands of dollars to run. And then, you find out something crazy. You show them an image and you ask, "What does this image depict?" And you will be surprised to hear the answer. It doesn't know. Yes, it cannot see. Now, what if I told you that they promised you a model that is 99% smaller, a speck of dust comparatively, and yet this tiny guy can see? That sounds impossible. Yes, maybe in our dreams, wishful thinking. But, it actually exists. It is made by DeepMind and is called Gemma 4. This runs on your laptop. It is an absolute gem. Free and
open, downloaded more than 300 million times by us fellow scholars. That is insane. I love it. And then, something amazing happened. Now, hold on to your papers, fellow scholars, because, yep, they gave us the secret sauce. They finally told us the architecture they used for Gemma 4 to see. And it's kind of crazy. So, we finally understand how this can pull off things so easily, like talking as a medieval bard while identifying objects in your video, and so much more. So, how did they add vision and multimodal reasoning into an unusually small local model? The secrets are finally out. Dear fellow scholars, this is Two Minute Papers with Dr.
Károly Zsolnai-Fehér. So, a conventional AI is really several neural networks connected together. And if you have an image, you need a dedicated visual model just for that. Or Okay, what about audio? Giving it ears. Yep, that needs an audio encoder. A specialized part for each of these tasks. But this is not
Direct patch projection unifies perception and reasoning
1 Highlight · Replacing specialized encoders with direct transformer projection
really looking or listening. This is just passing a translation around that is created by a different neural network. Their smallest models do that, but when we upgrade to the 12 billion model, things get crazy. Scientists at DeepMind say, "Throw that all away. Out. Right now." Instead, mhm, it cuts your picture into small patches. Then it projects those pixels directly into the model's internal representation. So, it knows where each patch came from. Okay, but what is the point here? Why do that?
Well, you don't need a separate neural network, a vision transformer, to interpret the image for you. No. Throw it out. Same for audio. Slice it up into 40-ms chunks, and then comes the magic. You just pour all these tokens into your main transformer, and then what happens? Well, this system is forced to learn to be the eyes, ears, and brain at the same time. And I think this is one of the most important architectural ideas in Gemma 4. It removes hundreds of millions
of specialist parameters, and it blurs the boundary between perception and thinking. And the result is an AI system that punches way above its weight. It handles images, it handles audio, and it is bloody smart. And this is something that you can own. Now, two more points.
Open-weight models and GPU infrastructure drive research
One, the Gemma 4 ecosystem continues to get improvements to make it faster and better. I'll give you a link to So in the description. Two, they shared the secret sauce there. So Gemma 4 is not just amazing in and of itself, but it can help Deep Seek and other systems learn to see better and more efficiently. That is absolutely amazing. What a time to be alive. And please do not take it for granted that these amazing open models will just keep coming in the future. These are gifts to all of us and these gifts may stop coming as capabilities increase. It is not a law of nature that we just get these models for free in the future, too. So, to everyone who is working on open models, wherever you are in the world, you are heroes. You help scientists, students, and millions of other people to do their work better.
Thank you so much. And we, as a community, have to come together and do everything we can to support these open systems. I use Lambda to reproduce AI research papers often in minutes. It's also great to train your own models or fine-tune an existing one. Run inference or text-to-image or video, easy-peasy. Running a Deep Seek chatbot or agent, super fast, super reliable. Lambda gives you powerful Nvidia GPUs to run your own experiments. I test ideas from the papers I cover and moments later, results. Love it. Seriously, try it out now at lambda.ai/papers.