Microsoft : How audible wayfinder app helps blind and low-vision users navigate public spaces

Lost and found

RightHear is one of a new generation of digital accessibility aids that are harnessing connected technology to improve the lives of the 1 billion people the World Bank estimates live with at least one disability.

A part of Microsoft's AI for Good Acceleration Program, which helps startups in Israel advance their artificial intelligence (AI) solutions and helped fast-track the app's development and commercialization, the app can be programmed for 26 languages. It has been built to integrate with other assistive technologies including Microsoft's Seeing AI. Right Hear co-developer Idan Meir expects to add connectivity with other apps soon, including navigational systems to guide users safely in from the street.

Raz Bachar, Microsoft for Startups managing director in Israel, brought RightHear into the program which provides support, including mentoring, business and technical advice as well as resources to prepare entrants' products for market. Bachar saw the potential of the app after it was awarded Geektime's Bootstrap Startup of the Year Award, which he describes as the "Oscars of startups."

"There are so many amazing startups in Israel but the one that makes you want to go above and beyond in supporting them is the one with founders who have a passion for the services they provide, the ones who are interactive with you - this is what we saw in Right Hear," Bachar says.

"They were not only able to grab the opportunity of the AI for Good program, they were also able to leverage our AI Azure Cognitive Services," he adds. "They have a unique solution to a great accessibility need, which aligns with Microsoft's investment in the accessibility ecosystem."

Companies are taking advantage of this adaptability to link into their own digital accessibility platforms. McDonald's, for instance, has installed RightHear at its 200 restaurants in Israel. It ties into other in-store visual aids such as magnification features on order terminals that have "low-view" displays that customers in wheelchairs can access. RightHear indicates where registers, bathrooms and other services are located.

"As a company whose orders are mostly made by digital platforms, we have been impressed by the smart technology that is simple and user-friendly," says Eyal Padan, head of digital marketing for McDonald's in Israel.

RightHear was built after consultation with people who are blind or have low vision, but the initial spark came from a different challenge faced by Meir. He and business partner Gil Elgrably had come across the iBeacon technology that gives RightHear its way-finding capabilities seven years ago but weren't sure how it could be beneficially used. While exploring ideas, they tested the beacons' usefulness in a variety of public spaces. But they wasted much of their time trying to find one another first.

"I always get lost, wherever I go - that's part of my life," Meir explains with an embarrassed laugh. "Gil would be on one side of a mall and I on another side and it would take an hour just to meet each other."

Meir's ability to get lost in even his hometown got him thinking about the difficulties faced by people who live permanently with a sense of dislocation about their immediate surroundings. Meir and Elgrably, who are also based in Ra'anana, didn't set out to create an accessibility app and initially considered incorporating the technology into a shopping coupon app, but that concept lacked something for Meir. As tech-minded entrepreneurs, they simply wanted to get a new product up and running.

The experience of developing RightHear has been a humbling one for them, illuminating issues surrounding orientation they hadn't before considered. That was underlined when the COVID-19 pandemic made close contact with other people potentially life-threatening.

"We were looking to do something more meaningful, more purposeful, more impactful," says Meir, a 34-year-old father of two.

The pair contacted disability groups in Ra'anana to understand how the technology could be best deployed and were connected with members of the blind and low vision community in their local area. Sarnetzky was among them.

"Gil called me and there is no other way to describe it than he interrogated me," she jokes. "He interrogated me for an hour-and-a half. What would I prefer - distances in meters or in feet? What would be more convenient for me - if we counted the steps or we didn't count the steps? They applied every single thing I suggested."

From that research a prototype system was built on Microsoft Azure and piloted in a local museum. Beacons were dotted throughout the facility, networked and programmed to highlight points of interest and important services via a digital control dashboard. Test users were then asked to try it out. The feedback was not just positive, it was also imbued with emotional significance.

The artificial intelligence behind the technology worked well. When the matchbox-sized beacons were placed at strategic positions they would connect to create an augmented reality within which physical things could be located. Users could log in and receive audible orientation prompts via regular headsets.

"The first reaction from users was really, really meaningful," Meir recounts. "It felt like I had given them a piece of the moon!"

Learn more about RightHear's origin story from its co-founder, Idan Meir: