Microsoft announced a new “multimodal” Phi-3 model capable of handling audio, video, and text at its annual developer conference, Build, this week. That came just days after OpenAI and Google both touted radical new AI assistants built on top of multimodal models accessed via the cloud.
Microsoft’s Lilliputian family of AI models suggest it’s becoming possible to build all kinds of handy AI apps that don’t depend on the cloud. That could open up new use cases, by allowing them to be more responsive or private. (Offline algorithms are a key piece of the Recall feature Microsoft announced that uses AI to make everything you ever did on your PC searchable.)
But the Phi family also reveals something about the nature of modern AI. Sébastien Bubeck, a researcher at Microsoft involved with the project, tells me the models were built to test whether being more selective about what an AI system is trained on could provide a way to fine-tune its abilities.
The large language models like OpenAI’s GPT-4 or Google’s Gemini that power chatbots and other services are typically spoon-fed huge gobs of text siphoned from books, websites, and just about any other accessible source. Although it’s raised legal questions, OpenAI and others have found that increasing the amount of text fed to these models, and the amount of computer power used to train them, can unlock new capabilities.
Bubeck, who is interested in the nature of the “intelligence” exhibited by language models, decided to see if carefully curating the data fed to a model could improve its abilities without having to balloon its training data.
Last September, his team took a model roughly one-17th the size of OpenAI’s GPT-3.5, trained it on “textbook quality” synthetic data generated by a larger AI model, including factoids from specific domains including programming. The resulting model displayed surprising abilities for its size. “Lo and behold, what we observed is that we were able to beat GPT-3.5 at coding using this technique,” he says. “That was really surprising to us.”
Bubeck’s group at Microsoft has made other discoveries using this approach. One experiment showed that feeding an extra-tiny model children’s stories allowed it to produce consistently coherent output, even though AI programs of this size typically produce gibberish when trained the conventional way. Once again, the result suggests you can make seemingly underpowered AI software useful if you educate it with the right material.
Bubeck says these results seem to indicate that making future AI systems smarter will require more than just scaling them up to still greater sizes. And it also seems likely that scaled-down models like Phi-3 will be an important feature of the future of computing. Running AI models “locally” on a smartphone, laptop, or PC reduces the latency or outages that can occur when queries have to be fed into the cloud. It guarantees that your data stays on your device and could unlock entirely new use cases for AI not possible under the cloud-centric model, such as AI apps deeply integrated into a device’s operating system.
Apple is widely expected to unveil its long-awaited AI strategy at its WWDC conference next month, and it has previously boasted that its custom hardware and software allows machine learning to happen locally on its devices. Rather than go toe-to-toe with OpenAI and Google in building ever more enormous cloud AI models, it might think different by focusing on shrinking AI down to fit into its customers’ pockets.
