After the well-known paper got here out, advances in generative AI basis fashions have centered round making the “transformer”-based fashions bigger and bigger. Instead of doing that, Sakana AI will give attention to creating new architectures for basis fashions, Jones instructed Reuters.

“Rather than building one huge model that sucks all this data, our approach could be using a large number of smaller models, each with their own unique advantage and smaller data set, and having these models communicate and work with each other to solve a problem,” stated Ha, although he clarified this was simply an concept.

All the authors on the “Attention Is All You Need” paper have now left Google. The authors’ new ventures have attracted thousands and thousands in funding from enterprise buyers, together with Noam Shazeer, who’s working AI chatbot startup Character.AI, and Aidan Gomez, who based giant language mannequin startup Cohere.