Facebook researchers are investigating AR glasses featuring “hear-through” technology powered by specialized in-ear monitors for “enhanced hearing.”

The technology “would be able to recognize different types of events happening around you: people having conversations, the air conditioning noise, dishes and silverware clanking. Then using contextualized AI, your AR glasses would be able to make smart decisions, like removing the distracting background noise — and you’d be no more aware of the assistance than of a prescription improving your vision,” according to Facebook.

The work with beamforming, adaptive noise cancellation, and machine learning is described as “an area we’re just starting to explore” with the goal of enabling “perceptual superpowers” like “enhanced hearing.” One of the possible paths would use the “pattern of your head and eye movements” to “automatically enhance the sounds you want to hear and dim unwanted background noise.”

Safe Use Questions

Michael Abrash, chief scientist at Facebook Reality Labs Research, responded to questions in a briefing call with journalists related to how the use of this technology might affect social norms. Could a personal conversation at a restaurant table be heard by a nearby AR-enhanced patron? Would the nearby patrons ever know their conversation was heard by someone else? And would that conversation be less likely to happen in the first place if people knew mediated hearing was more common than it is at the time of this writing?

Abrash and Ravish Mehra, the audio team lead at FRL Research, gave a few examples of some potential mitigation strategies that might be employed to limit the range of this feature in the future. Mehra explained in a prepared statement they intend “to put guardrails around our innovation to do it responsibly, so we’re already thinking about potential safeguards we can put in place…for example, before I can enhance someone’s voice, there could be a protocol in place that my glasses can follow to ask someone else’s glasses for permission.”

Another line of research at Facebook may explore the use of personalizing audio with a head-related transfer function (HRTF) which accounts for the individual shape of your ears, potentially using an “algorithm that can approximate a workable personalized HRTF from something as simple as a photograph of their ears.” A Facebook representative added “the audio research team is considering several novel approaches to scaling the capture of people’s unique HRTFs.”

A Facebook blog post explains:

“Another issue the team is keenly aware of is the capture of sensitive ear data, both in the research phase and beyond. Today, before any data we collect is made available to researchers, it is encrypted and the research participant’s identity is separated from the data such that it is unknown to the researchers using the data. Once collected, it’s stored on secure internal servers that are accessible only to a small number of researchers with express permission to use it. The team also has regular reviews with privacy, security, and IT experts to make sure they’re following protocol and implementing the appropriate safeguards…”

AR vs. VR And In-Ear vs. Open-Ear

Mehra explained in the call that different audio-based modifications might be accomplished in future AR or VR-based sound delivery systems depending whether they feature in-ear or open-ear designs.

An open-ear audio design like the Valve Index or Oculus Quest, for instance, wouldn’t necessarily work well for the cancellation of sounds from the physical world, the researchers explained.

In-ear designs like Apple’s AirPods Pro might end up being a better fit for some use cases. They already feature “Active Noise Cancellation” and “Transparency mode” to switch between “depending on how much of your surroundings you want to hear.” Another in-ear design, Google’s Pixel Buds 2, offers an experimental feature called “attention alerts” that would lower volume when they detect a baby crying, a dog barking, or an emergency vehicle siren.

Facebook researchers, meanwhile, are using “prototype in-ear monitors” or “IEMs” which they say “can deliver the full experience of auditory superpowers. This lets us enhance the right sounds for you and dim others, making sure that what you really want to hear is clear even in loud background noise.”

“Our IEMs also feature perceptually transparent hear-through,” Audio Experiences Lead Scott Selfon explained in a prepared statement, “making it sound like I have nothing in my ears, and letting me safely hear the entire world around me.”

When asked if this same research could be applied to dynamically lower the sound from a VR headset to better hear someone talking to you in the same physical room, Abrash said he hadn’t thought of that before but it’s a “great idea.”

Future Aims

Facebook aims to make “stylish AR glasses” and pitches the research as being “focused on transforming communication for everyone, everywhere” while citing Johns Hopkins research suggesting one in five people have hearing loss and many “don’t use hearing aids for a variety of reasons including expense, social stigma, discomfort, and lack of reliability.”

“I’ve been wearing hearing aids since I was a little girl,” Technical Program Manager Amanda Barry said, in a prepared statement. “The ability to help people stay connected with their families as they get older and their hearing fades — that’s really pretty exciting.”

Facebook is also researching what it calls LiveMaps (an updating map of the physical world “with shared and private components”) and Codec Avatars (hyper-realistic personalized representations of human bodies) and either or both might be used in conjunction with the audio research. Overall, Facebook researchers seem set on a path to “defy distance” by providing “true social presence.”

“The only reason we need for virtual sound to be made real is so that I can put a virtual person in front of me and have a social interaction with them that’s as if they were really there,” Facebook Research Lead Philip Robinson explained in a prepared statement. “And remote or in person, if we can improve communication even a little bit, it would really enable deeper and more impactful social connections.”

Facebook recently launched a beta for its Horizon social network and added a new requirement that all future Facebook VR headsets be connected to a Facebook account. Starting in October, Facebook will be implementing new terms of service which state that “when people stand behind their opinions and actions, our community is safer and more accountable. For this reason, you must: Use the same name that you use in everyday life” when you’re using Facebook.