Background
Prior to the production release of the first generation Hololens AR device, visionaries at Stanford medicine wondered if the headset could be used during surgery to improve outcomes. At that time, one quarter of women who had a breast lumpectomy to treat early-stage breast cancer in the United States, had repeat surgery due to concerns that residual tumor was left behind. Stanford medicine and Microsoft decided to collaborate to see if they could indeed reduce the need for return procedures.
IMAGINATION: From Potential to Prototype with AR Medical Imaging
When Microsoft’s HoloLens team set out to create a prototype that would allow surgeons to view MRI data on breast cancer patients in 3D during surgery, they knew they needed a partner that could bring some serious technical chops to the table. Looking for an alternate perspective, Microsoft sought outside help to make their vision a reality.
But they weren’t just looking for any development team—they needed a group of experts who could create an augmented reality (AR) application capable of a consistent precision achievable by surgeons regardless of where they were standing when they used the headset. It was a tall order, to say the least. But with the help of Oscar Murillo, Design Director at Microsoft on the HoloLens team, they were determined to make it happen because they knew this application had the potential to positively disrupt the medical industry in ways that could be truly revolutionary for surgeons and patients.
CREATION: Inspiring Innovation Through Collaboration
Microsoft was on the hunt for a partner to ignite creativity through collaboration with the internal team. They knew they needed more than a bunch of developers. They needed visionaries who would not be constrained by conventional assumptions and boundaries or afraid to to color outside of the lines. They need experts who were passionate about the opportunity this technology offered and who would work closely with the brilliant surgeons at Stanford medicine to create something truly groundbreaking. Enter the Chaos Theory team, the perfect match for Microsoft’s dream team.
Together, we rolled up our sleeves and got to work, leading a team of HoloLens experts, the lead oncologist at Stanford medicine, and a host of other specialized partners through our integrative five-step process. With our help, Microsoft’s campus was transformed into a hive of innovation, as we all worked side-by-side to bring this revolutionary application to life.
Imagine
We kicked off our mission with collaborative ideation, diving deep into research, developing personas, and ultimately crafting a design brief that was on point with the project goals. But we didn’t do it alone. Our close partnership with Stanford medicine helped us more deeply understand both the surgeons who would use the application as well as the patients who would benefit, so that the personas we created were truly spot-on.
Design
We kept the momentum going with our combined powerhouse team, diving headfirst into the creation of product requirements, solution architecture, and functional prototypes for an AR application for the MS HoloLens. By working together synergistically, our collective expertise began to unveil the magic that would lead to success.
Develop
Each part of the team – Microsoft, Stanford medicine, and Chaos Theory – brought knowledge to the partnership that helped us work together to craft a story map that brought visibility to the core features that were required. Together, using an agile sprint cadence, we forged the path to creating a prototype that would be a game-changer in the surgical world.
Deliver
Performance testing and optimization is a part of any good delivery and, for this aspect of the project, we got to do this working alongside Stanford surgeons. We immersed ourselves in making adjustments to improve the hologram accuracy in all dimensions which resulted in the delivery of solid documentation that Microsoft would build on for future innovation.
Deploy
Beyond design, our human-centered approach extended to unwavering client support. We stayed available to the Microsoft HoloLens team as they developed further around the AR application and as the application was later adopted into use by Stanford medicine.
Technology Used:
Unity 3D development platform
MRToolkit
Microsoft HoloLens development kit
What’s it like working with the Chaos Theory team?
Infusing Collaboration with Excitement . . .
“The process they used is a phenomenal way of coming up with more innovative solutions and does it in a way that feels more collaborative and participatory. You get a lot more energy and excitement around it.” And Oscar definitely appreciated the energy that Chaos Theory brought to this important collaboration. “The Chaos Theory team jumped into this project with the same level of excitement that a four year old will jump at a beach full of sand on a sunny day.”
. . . While Keeping it Effortlessly Efficient
Oscar was impressed with our team’s knowledge and work ethic. “Chaos Theory thoroughly understands the business experience and technology challenges. They bring incredible energy and passion. And they are incredibly efficient in getting to a great solution quickly.”
INNOVATION: Improving Surgical Outcomes for Patients
Big ideas require big thinking and collaborative partnerships to see them reach their potential. Through this diverse partnership, we all helped to create the first visualization of MRI data within the Microsoft HoloLens AR system.
By giving surgeons the power to visualize data directly on the patient, as well as the size and location of the tumor in three dimensions, the AR application is the first step toward increased accuracy in removing tumors and a reduction in followup surgeries.
Technically speaking: the augmented reality (AR) prototype was built using a mixed reality (MR) toolkit and the Unity 3D development platform. The input was patient MRI data and the output was the 3D hologram projected on the patient’s body through the Microsoft HoloLens headset.
A paper from Stanford medicine Radiology and Oncology fellows was published in IEEE and stated, “The first results from breast cancer surgeries have shown that mixed-reality guidance can indeed provide information about tumor location, and that this exciting new use for AR has the potential to improve the lives of many patients.” Work continues today to bring more mixed-reality solutions into medicine to improve patient outcomes.
Our work enabled continued surgical innovation
For Oscar, he was pleasantly shocked that the team was able to deliver a solid solution when the HoloLens product itself was still at such an early stage. “The work that Chaos Theory did was exceptional. They were able to essentially create a sustainable solution when we were still building out the development toolkits. Basically, they built a plane, mid-flight, in the middle of a hurricane.”
Relevant case studies
Interactive Experience
UNICEF Giga Project
UNICEF knew that the multitude of challenges in trying to create a connected world would require a solid data management foundation. The Project Giga data science team turned to Chaos Theory for the solution.
Transit
Community Transit
Community Transit, a public agency serving 5 million passengers annually in Snohomish County, Washington, realized that they needed to revamp their user experience to stay relevant in the digital age. Chaos Theory engineered a new suite of transit information tools to serve their users.
Interactive Design
Seattle Art Museum
The Seattle Art Museum (SAM) wanted to do more than simply reflect diversity in their American art exhibit – they wanted a way to engage the diverse breadth of visitors in the exhibit itself. Chaos Theory worked with SAM to create a unique and immersive interactive experience.