When Meta released Llama 2, a powerful artificial intelligence model similar to the one behind ChatGPT, last month, it made it possible for developers, startups, and researchers to play with the kind of AI that has enthralled the world for nearly a year.
Today, Meta is following up with the release of Code Llama, a version of the model that has been tuned for programming tasks. The release could mean more developers getting a taste of AI-assisted coding. It could also inspire new ways of embedding AI into software. And it could help further establish Meta as the preeminent supplier of “open” AI tools.
“It’s exciting that they’re releasing the weights to the community,” says Deepak Kumar, a postdoctoral researcher at Stanford who has studied AI coding, referring to the parameters of the neural network at the core of the model.
Kumar says the release of Meta’s regular language model Llama 2 led to the formation of communities dedicated to discussing how it behaves and how it can be modified. “It gives us a little bit more flexibility to play with what exactly is going on under the hood, compared to these closed-source models from Google or OpenAI.”
Kumar says developers are likely to build new kinds of applications using Code Llama. For example, it could be possible to create a programming assistant that performs various additional safety checks before recommending a chunk of code, says Kumar, whose own research has explored how AI assistance can sometimes lead to less secure code. Kumar adds that the release could inspire the creation of assistants specialized for particular kinds of coding.“You can build all sorts of tooling on top of the model,” he says.
Talia Ringer, an assistant professor at the University of Illinois Urbana-Champaign who researches programming, says that Code Llama will be valuable for academic research. I already have students using Llama models for research, and I could see those students being extra excited about a code model given the nature of our work,” she says. But Ringer adds that, ideally, the data used for training would also be released. “That’s often the missing piece for making sense of research on LLMs,” she says.
Programming is one area in which recent advances in AI have already had a considerable impact.
In May 2021, GitHub, a subsidiary of Microsoft, launched Copilot, a plug-in for coding programs that auto-completes sections of code based on the first line or a comment typed by the user. Copilot uses a version of Open AI’s GPT, the large language model behind ChatGPT. That model is trained further using code that GitHub stores for developers, as well as, reportedly, by contractors who are paid to annotate their own code.
GitHub faces a lawsuit for using some open source code in its training data, and Masad says Meta is likely to have limited the training data to avoid such complications. Copilot costs $10 per month for individuals and $19 per month, per user, for businesses.
Copilot has apparently been a hit with developers. According to figures released by GitHub in June, it is used by more than over a million developers and more than 200,000 businesses. The company’s own studies also suggest that Copilot accelerates the rate at which coders can perform tasks, leading to a 30 percent increase in productivity.
Meta is releasing two versions of Code Llama, one geared toward producing Python code and another optimized for turning natural language commands into code. It is also making three sizes of models available. The smallest can run on a single GPU.
Meta says that Code Llama is trained on code that is in the public domain. In two common coding benchmarks, HumanEval and Mostly Basic Python Problems, it performs much better than existing open source coding models and is “on par with ChatGPT,” the company says.
Amjad Masad, CEO of Replit, an online coding platform that offers several generative AI tools, does not expect Code Llama to supplant Copilot because it’s more limited training data is likely to make it more limited. But he says the release could allow developers to experiment with agents that perform useful tasks, like browsing the web for information or using an API to book a flight or order a meal. “I think that’s a really exciting area,” Masad says. “Interactions where you can type natural language instructions, and the model can crunch data can do interesting things in the world.”
The release of Code Llama may also provide benefits to Meta. The company might not have ChatGPT or an AI-powered search engine, but establishing itself as the provider of free AI to many developers, companies, and researchers could give it a foothold in the race to harness generative AI. Meta chose to adopt an open approach after seeing someone leak an early version of Llama to the web in May.
Neither Llama 2 nor Code Llama are not released under regular open source software licenses that would allow unfettered commercial usage. Under Meta’s license, for instance, users are restricted from using the models in apps or services with more than 700 million monthly users.
A research paper posted online this month notes that releasing AI tools can have significant indirect benefits to the companies behind them by locking researchers into their tools, for instance, and providing them with new ideas that they can use at scale.
