Earlier this month, entrepreneur Corey Jaskolski pulled out a pen and drew his greatest guess at what the surveillance balloon shot down by a US jet would have seemed like from area. Then he fed the sketch and “a gob” of current satellite photos from the world the place the balloon was taken down into algorithms developed by his picture and video detection startup Synthetatic, and waited.

Within two minutes, he says, the algorithms discovered the 200-foot-tall balloon off the coast of South Carolina. “I couldn’t believe it,” Jaskolski says. Nor may his spouse when he excitedly confirmed her his outcomes. But when he estimated the altitude of the balloon within the picture it was round 57,000 toes—matching the peak at which the balloon was spotted by a US spy plane—and social media sightings from 20 minutes earlier than the picture was taken appeared to verify he had discovered it.

Jaskolski dug in, poring over wind fashions and social media sightings to feed his software program, known as RAIC (fast computerized picture categorization), new swathes of satellite tv for pc information from the corporate Planet Labs. The device is designed to make it attainable to look giant picture collections for objects of curiosity utilizing a single instance picture.

“We drew a big arc across time and space and started searching that,” Jaskolski says. Having discovered the balloon as soon as, Synthetiatic’s software program might be skilled with an actual picture of the balloon to additional information its search.

Over the subsequent a number of days, Jaskolski put RAIC to work. The firm has since compiled six sightings of the balloon (5 confirmed, one nonetheless being investigated) on its satellite tv for pc imagery and has used wind information to estimate the way it moved between these factors. “We can draw a 1-kilometer-wide track across the whole of the United States and just follow the balloon,” he says. “We have a track from where it entered from Canada, all the way to South Carolina, where it got popped, with six points along that arc.”

Jaskolski’s stratospheric scavenger hunt might have been made attainable by sensible software program, but it surely additionally required human professional information. His preliminary drawing of the craft seemed extra like a technicolor snowman—stacked crimson, inexperienced, and blue circles. The goal was to imitate the best way satellites typically seize totally different wavelengths of sunshine utilizing separate sensors that aren’t at all times synced in time, creating a number of disjointed views of objects. And it throws up false positives.

But the flexibility to map a surveillance balloon’s path with such readability might be a sport changer for national security, says Arthur Holland Michel, senior fellow on the Carnegie Council and author of a book on drones and surveillance. “The combination of AI with satellite imagery is undoubtedly a very powerful technology for surveillance and espionage and counterespionage,” he says. 

Holland Michel additionally factors out that satellite tv for pc imagery and AI have their limitations. The methodology by which Synthetatic first discovered the balloon—utilizing a drawing—may end in false positives if the thing of curiosity was one thing extra complicated or much less publicly documented, comparable to a tank. “Things often look a bit weird and unfamiliar from above,” he says.

“There’s undoubted potential there,” Holland Michel says, “but it’s easy to think this combination of satellites and AI is an all-seeing capability that will lay everything bare.” It’s helpful in sure circumstances, just like the balloon, he says, however probably not all situations.

That’s one thing Jaskolski acknowledges—however he additionally considers the mission an instance of how human experience and grunt work will be elevated by AI. “This human-machine collaboration is my idea of how AI works today,” he says. “And it’s definitely how we build our product.” The device is presently used for humanitarian functions, together with by the UN World Food Program to find flood victims.

The pursuit of the balloon isn’t over simply because Jaskolski has managed to trace it throughout the United States. He says the method is “resource-intensive” as a result of the software program isn’t excellent and turns up many potential sightings that need to be whittled down by folks. “But we’d like to still continue to track it,” he says. “Whether we go all the way back to China or not, we feel like we solved a technical problem at least. We’d be crazy not to try.”