VR/AR Challenge finalist WayPoint RX take the guess work out of filling prescriptions

We received 67 submissions to our first ever AT&T VR/AR Challenge. After much deliberation, we’ve narrowed down the finalists (and one alternative) and will fly them to Las Vegas to demo their solutions at the 2017 AT&T Developer Summit and compete for a grand prize of $20,000. Here’s your chance to learn about the finalists as we introduce each one in a dedicated blog post.

Team Waypoint believes that the true potential of Mixed Reality is improving human health and wellbeing by placing the knowledge and experience of experts into the hands of everyday people. In fact, according to the FDA, pharmacy mistakes result in more than one death per day and over one million medical injuries per year. Medical techs can often fill up to 150 prescriptions a day. After speaking with several healthcare professionals, we learned that most pharmaceutical mistakes arise due to unfamiliarity with drug names, pill appearance, dosage, and medication location within pharmacies. These issues are compounded by the high-stakes and stressful nature of pharmacy jobs.

With Waypoint Rx, we use mixed reality technology to empower non-pharmacists to fill prescriptions quickly with no errors. The team’s prototype is an augmented reality (AR) app that allows a layperson to fill a prescription that otherwise could only have been filled by a highly trained pharmacist. The system maps a physical pharmacy to its digital representation. Upon receiving a prescription order, the prototype dynamically generates and displays the optimal path to fill the order based on the medication location and pharmacy layout. Holographic waypoint markers highlight the exact location of the medication and provide contextual information, such as amount needed and a 3D model that allows users to confirm they are collecting exactly the right items. With this guidance, users do not need prior knowledge of the pharmacy environment or medications and can quickly and accurately fill even a complex prescription order.

Through this type of rich contextual awareness, WayPoint Rx significantly reduces pharmacy errors by giving any hospital employee the knowledge and expertise of a trained pharmacist. If adopted widely, this system has the potential to save hundreds of thousands of lives. We caught up with team members Umar Arshad, Varun Mani, and Sara Remsen based in Boston, MA, Jan-Erik Steel of Old Chatham, NY, and Paul Katsen of San Francisco, CA to learn more about the idea for WayPoint RX and their interest in mixed reality.

Q: How did you come up with the idea for your project?

A: We held an ideation session where we discussed various problems that could only be improved through the application of augmented reality (AR). Although we identified several potentially valuable applications, we were most interested in applications that created a strong positive impact on individual lives. Umar’s wife is a surgeon and had often complained about problems related to obtaining medications during emergency situations. Other team members had also dealt with pharmacy errors in various situations. Realizing that this was a potentially underserved but important problem that was uniquely suited for AR, we decided to work on addressing it.

Q: Tell us a little bit about your team and how you worked together to create your entry:

A: We first met in October 2016 during the Reality, Virtually Hackathon (the biggest AR/VR hackathon in the world); realizing that each of us had very complementary skills, we decided to form a team. Over the course of the ensuing 50 hours, we worked almost telepathically to learn completely new programming languages, UX paradigms, game engines, and even design skills in order to create a groundbreaking (and functional) AR application designed to help pharmacies fill prescriptions accurately and quickly. Ultimately our prototype won two prizes at the hackathon, including the grand prize for AR. After realizing how amazing it was to work with each other—particularly in a high-stress, high-pressure environment where we were all learning something completely new—we knew we needed to form a startup together.

Q: What does the AT&T Developer Program and contests like this mean to you?

A: Developing AR applications is a unique task that requires a lot of feedback from user testing. Obtaining this feedback is often very difficult. However, getting a chance to demo our prototype to a large group of people provides an incredible opportunity for us to test our work and gain feedback in a short amount of time. In addition, as a young company it is always helpful to get support from larger companies such as AT&T, as well as get the chance to earn award money to fund further experimentation and development of our prototype.

Q: Why should the audience vote for your project?

A: Our demo should clearly indicate the value that heads-up AR guidance can provide to non-experts in any industry. Not only does our prototype allow more people to do the same work that could only have been done by a small set of highly-trained experts, it also allows them to do so at a speed and accuracy that far surpasses anything that could be done without AR. Most importantly, the prototype showcases the ability for AR to be used to keep humans at the center of the workforce of the future. It also demonstrates the possibility for the technology to be used to save hundreds of thousands of lives.

Q: What advice do you have for new developers just getting started in the field?

A: Developing AR applications requires combining a set of different skills that previously were never used together. These skills include UI/UX design, familiarity with 3D graphics and physics engines, computer vision skills, and physical workflow/process design. Moreover, not only do AR applications require putting these skills together in a functional way, they require that the application be highly intuitive to use because most people have no pre-existing experience with AR.

Q: What is the development background of all the team members?

A: We are a cross-disciplinary team of individuals with different and highly complementary skills that give us a competitive edge in building AR applications. We bring a wealth of experiences and expertise to the team, but we share a common vision: using AR to solve human problems. Along with our business expertise, we also have a deep bench of experience working on interactive 3D UI and enterprise technologies.

• Varun is an incredibly skilled AR technologist and current student at Harvard Business School, and was previously a member of the core team that developed the ground-breaking HoloLens headset at Microsoft.

• Sara is a talented technologist and current IDM Fellow at MIT with design and development experience in 3D animation and education, as well as previous experience at early-stage startups.

• Umar is a recovering lawyer, technologist, and current EMBA student at MIT, with over 14 years of experience advising some of the world’s most innovative companies on their most cutting edge UI and UX products.

• Jan-Erik has over 15 years of experience leading the design and development of cutting-edge video games such as Guitar Hero and Skylanders, giving him a unique expertise in building interactive software that is easy and intuitive to use.

• Paul is an accomplished full-stack technologist and the CEO and co-founder of a YC and venture-backed B2B enterprise software company.

We strongly believe our interlocking skills give us a distinct advantage in AR development, and allows us to move and build at a pace that is difficult to replicate.

Q: What technologies are you most excited about?

A: We are most excited about head-mounted AR with spatial awareness. Heads-up display (HUD) technology has existed for a while, but the true breakthrough of new AR hardware is the ability to map a 3D digital world onto the real world using depth sensors and computer vision. These innovations now open a whole new realm of possible applications!

Q: What are your future plans for your project?

A: We are planning on launching a startup to apply our experience with Waypoint RX to the larger problems faced by the order fulfillment industry.

Q: If you win, what do you hope to do with the prize money?

A: We plan on using it to continue developing our prototype and to buy another HoloLens dev kit.

What do you think of Waypoint RX? Do you think it change the fulfillment? Let us know in the comments.