Can Coding Automation Companies Deliver a kind of SWE AGI?
Is it going to be like Magic, really?
Hey Everyone,
After a summer break we’re back to our usual frequency here. Welcome back! I really wanted to cover Magic, their recent funding and their partnership with Google that lacked really transparent disclosures.
For Software engineers though I think it might be important.
Apparently it's rather lucrative to state that your goal is AGI. Among all the coding AI automation startups, this one is the loudest on my radar. They were working on larger (I mean very large) context windows before Google made their breakthroughs, okay improvements.
Who dares speak against the magic of this Generative AI land? Hmm, these kids must be pretty special. They literally just announced in a blog post on Thursday, that they closed a $320 million fundraising round with contributions from Schmidt, as well as Alphabet’s CapitalG, Atlassian, Elad Gil, Jane Street, Nat Friedman and Daniel Gross, Sequoia and others.
On August, 29th, they released LTM-2-Mini Magic's first model with a 100 million token context window. That’s 10 million lines of code, or 750 novels. They will need a lot of compute so these latest funds will be used to expand Magic's AI-driven coding tools and build two supercomputers on Google Cloud. These supercomputers, named Magic-G4 and Magic G5, will utilize Nvidia H100 GPUs and Nvidia’s next-gen Blackwell chips respectively.
Ultra Long Context Windows or Long-Term Memory (LTM) Models
Magic AI is a software development automation, targeting their own conception of AGI. Typically in the era of OpenAI, all AGI means are commercial tools capable of doing what a person could do better. As for their compute, they hope to scale these clusters to tens of thousands of GPUs over time, achieving 160 exaflops of computational power. They have now raised in area of what Inflection raised before they got acquired by Microsoft.
Instead of relying on fuzzy memorization, Magic AI's LTM (Long-Term Memory) models are trained to reason on up to 100M tokens of context given to them during inference. People like Eric Schmidt (video) seem to think long-context windows are a big deal.
Founded in 2022 by Eric Steinberger and Sebastian De Ro, Magic has rapidly attracted investor interest. Is it the long context windows or that they claim to be working on AGI, or is it something else?
With the latest raise, the total funding raised by Magic accounts for $465 million. The Coding AI assistants and automation space is a very lucrative niche in the Generative AI era proper (2017-2025). They are a lot of other promising names like Codeium, Cognition, Poolside, Anysphere (Cursor) and Augment, to name a few that sound similar to me.
Their Series A was in February, 2023 so they must have made a lot of progress to be already ready for the that kind of compute. In describing what they are trying to do: this iteration of AGI is pretty simple: Magic focuses on developing models that automate various aspects of software development, such as code generation, debugging, and project planning. Magic's AI-driven tools (not yet available for sale) are designed to help software engineers write, review, debug, and plan code changes. These tools function like an automated pair programmer, continuously learning and adapting to the context of different coding projects. Basically like a smarter Github Copilot, I think.
Startups like Cognition (Devin AI) and Anysphere (Cursor) seem either very well funded or fairly popular already among actual software engineers, so it's curious to see what Magic AI will be able to accomplish in the space. Outside of OpenAI, Magic really caught me off guard when they started promoting themselves under the AGI banner.
Their blog post about their ultra long context window was not compelling. But Eric Schmidt has so far invested in both Magic AI and Augment. AGI aside, what they both seem to focus on is a smarter kind of personal assistant that can code and do more automation with one foot in the Agentic AI bandwagon. I guess soon we'll all be augmented like magic. This is one of the key narratives of Generative AI of course, so we'll have to see if it actually comes true.
Frontier scale code AGI aside, Magic AI also want their own benchmark for things like their ultra long context window. Magic also introduced HashHop, a new benchmark for evaluating long-context models. This tool addresses limitations in current evaluation methods by using incompressible hash pairs, providing a more rigorous test of model capabilities.
In addition to the investment, Magic also announced a partnership with Google Cloud to construct two AI supercomputers. No mention of the terms of this deal or how Google is involved besides being their Cloud hosting and GPU host. The company has a small team of around two dozen people and is targeting a market that could be worth $27.17 billion by 2032.
The startup's focus on inference-time compute sets it apart in the competitive AI landscape. Magic believes that pre-training has limitations and that inference-time compute represents the next frontier in AI. In July we heard that Magic was seeking to raise over $200 million at a $1.5 billion valuation. It seems there was way more interest than even they expected in the round.
According to them, their LTM (Long Term Memory) mechanism needs >1,000x less compute and memory than Llama 3.1 405B’s attention. Llama 3.1 would need 638 H100s *per user* to store a 100M token KV cache. LTM needs a small fraction of one.
SSMs, RNNs, and RAG all exploit weaknesses in evals like Needle In A Haystack, so they made their new eval, HashHop:
1) Incompressible
2) Multi-hop
3) No semantic hints
4) No recency bias
Google is fairly interested in long context windows too with Gemini. Github Copilot is now a key driver for Microsoft's rather clever acquisition of Github, back in 2018. It has now nearly 2 million paying subscribers since its launch in 2022 and contributed to a 45% year-over-year increase in GitHub’s revenue. I believe one of the main benefits of Generative AI is the impact on code automation and productivity of software engineers, that can drive innovation faster and allow startups and Enterprise companies to do more with less, in the not so distant future.
AI assistants for writing and editing code, and several startups in this field raised a total of $906 million since January 2023, including $433 million so far in 2024. With that kind of scale, Magic AI (more correctly known as Magic Labs) is now suddenly a major player among those.
With Google backing here it's obvious that Magic’s focus is on ultra-long contexts, that could be a key to setting it apart in the industry.
“We are excited to partner with Google and Nvidia to build our next-gen AI supercomputer on Google Cloud,” Magic CEO and co-founder Eric Steinberger said in the post. “Nvidia’s GB200 NLV72 system will greatly improve inference and training efficiency for our models, and Google Cloud offers us the fastest timeline to scale, and a rich ecosystem of cloud services.”
Early investors were of course familiar names in Nat Friedman & Daniel Gross, Elad Gil, and CapitalG. CapitalG is of course Alphabet's independent growth fund. We also thought OpenAI Fund was independent, until we found out it was actually owned by Sam Altman. OpenAI then did an entire PR campaign about how Sam Altman was "removed from the Fund". I highly doubt that.
Just as a coding automation startup calling itself an AGI company is somewhat for marketing purposes. Their first seed round was in the summer of 2022. The familiar Angel names participating in this startup are CapitalG (Alphabet’s Independent Growth Fund), Nat Friedman (former CEO, GitHub & co-creator of Github Copilot), Elad Gil, Amplify Partners, Noam Brown (Research Scientist, Meta), Mehdi Ghissassi (Director of Product, Google Research), Bryan Pellegrino (LayerZero), Arthur Breitman (Tezos), 10x Founders, William Tunstall-Pedoe (Unlikely AI), Adam Jafer & Fredrik Hjelm (Voi), Klaudius Kalcher & Roland Boubela (MostlyAI), Xavier Sarras, Florian Huber, Daniel Dippold, and others.
Magic is a very small team and they build vertically integrated – they do research, train foundation models, build products, and talk to users. Their team should get fairly bigger now with the latest funding round. Major new investors in this round are Eric Schmidt, Jane Street, Sequoia and Atlassian.
Read their late August, 2024 blog to check them out and see all the new roles they are hiring for. You just got to believe in the Open spirit of AI of their era, it could be like magic ✨.