Runway has shouldered apart Midjourney and Stable Diffusion, introducing the primary clips of text-to-video AI artwork that the corporate says is totally generated by a textual content immediate.

The firm mentioned that it’s providing a waitlist to affix what it calls “Gen 2” of text-to-video AI, after providing an identical waitlist for its first, easier text-to-video instruments that use a real-world scene as a mannequin.

When AI art emerged last year, it used a text-to-image mannequin. A consumer would enter a textual content immediate describing the scene, and the device would try to create a picture utilizing what it knew of real-world “seeds,” inventive types and so forth. Services like Midjourney carry out these duties on a cloud server, whereas Stable Diffusion and Stable Horde make the most of comparable AI fashions working on residence PCs.

Text-to-video, nonetheless, is the subsequent step. There are varied methods of conducting this: Pollinations.ai has amassed a couple of fashions which you’ll check out, one of which merely takes a couple of associated scenes and constructs an animation stringing them collectively. Another simply creates a 3D model of an image and lets you zoom round.

Runway takes a special strategy. The firm already provides AI-powered video tools: inpainting to take away objects from a video (versus a picture), AI-powered bokeh, transcripts and subtitles, and extra. The first era of its text-to-video instruments allowed you to assemble a real-world scene, then use it as a mannequin to overlay a text-generated video on high of it. This is often achieved as a picture, the place you can take a photograph of a Golden Retriever and use AI to remodel the picture into a photograph of a Doberman, for instance.

That was Gen 1. Runway’s Gen 2, as the corporate tweeted, can use present photographs or movies as a base. But the expertise also can fully auto-generate a brief video clip from a textual content immediate and nothing extra.