liVeR_Banner_9.gif

Overview

liVeR | VR and Desktop

Enhancing surgical outcome and learning via medical VR.

I envisioned and implemented a suite of 3 applications (VR and Desktop) for enhancing HPB (liver, pancreas, gallbladder) surgical planning and education at Canada’s leading hospital, directing the UX process and collaborating cross-functionally.

    • Dr. Chaya Shwaartz, Director of HPB Center, Toronto General Hospital

    • Albert Fung, Team Lead, Toronto Video Atlas of Surgery

  • 16 months (concurrent to master’s projects)

  • Product Designer, Unity Developer, Project Lead

  • Master’s Capstone Project

    • UX Research: Interviews, Field Observations, Literature Review, Competitive Analysis, User Journey Maps, Personas, Affinity Diagrams, Usability Tests, Heuristic Evaluation

    • UX Design: Storyboards, Wireframes, Mockups, Functional Prototypes, Design System, Design Document, Information Architecture, User Flow

    • Strategy: Product Strategy, Stakeholder Alignment

    • Development: AI-Assisted Scripting

    • Project Management: Scope Definition, Timeline Creation and Management, Task Delegation

    • Development: Unity, Meta Interaction SDK, Normcore, GitHub, Visual Studio, ChatGPT

    • Design: Figma, Adobe Creative Cloud, Blender

    • Biomedical Communications Advisors: Dr. Derek Ng, Dr. Jodie Jenkinson

    • Technical and Medical Content Advisors: Albert Fung, Dr. Chaya Shwaartz

  • Surgeons and Surgical Trainees in the Toronto General Hospital HPB team

Feel free to keep scrolling to dive into the challenges and solutions, or take a quick look here at some key stats and highlights of what I did:

  • Incentivized 3.75x funding by solving key pain points.

  • Designed and developed 2 VR and 1 desktop applications.

  • Planned and conducted formal usability tests with 12 end users.

  • Used ChatGPT to script 4 functional prototypes and 3 final applications.

  • Produced 237 UI assets, 20 onboarding GIFs, and 1 device donning video.

  • Designed 1 logo and developed brand name.

  • Regularly assessed feasibility with Technical Advisor.

  • Led alignment meetings with clients to refine product vision.

Discover | Define | Design | Develop | Reflections

My process, being continuously adapted based on the dynamic context, consisted of discovery (research, strategy), define (scope), design (wireframes, functional prototypes, branding, usability tests, and Unity development), and delivery.

Discover

〰️

Discover 〰️

Problem Statement

How can we democratize 3D HPB (liver-pancreas-gallbladder) anatomy models to enhance surgical planning and education?

Dr. Smith, an experienced HPB surgeon in a globally-ranked hospital, has just wrapped up an exhausting 24-hour liver transplant. She has a flight to catch for a Berlin medical conference, but first, she’s back in her office, reviewing a new patient’s CT scans. Instead of exploring an intuitive 3D model to understand the complex anatomy of her patient’s liver, she must mentally reconstruct their organ from 64 flat CT slices on her computer.

Down the hall, Craig, a 4th-year med student preparing to observe this very surgery, is grappling with the same set of CT scans. Just starting to learn the CT viewing software, he is not yet confident with its antiquated navigation controls. After hours of struggling with complex software interactions and cross-referencing his 600-page anatomy atlas, he learns that the patient’s liver differs significantly from the textbook. Frustrated, he begins hand-sketching a 3D reconstruction in his notebook instead.

Research consistently shows that 3D HPB anatomy models enhance surgical outcomes and learning, but they are not the standard visualization for surgeons and trainees in North America. Why is this so, and how can we overcome these challenges?

Key Differentiators: Non-Digitally Native, Intimidated by New Tech

Key Differentiators: Excited about Integrating 3D, Digitally-Native

Secondary Persona

“Reducing ambiguity of 2D data via 3D models will benefit all levels of HPB surgeons.”

— Craig, the hopeful Medical Student

Primary Persona

“With so much on my plate, I’m simply disinclined to learn new technology with marginal benefits”

— Dr. Smith, the expert HPB surgeon

Pain Points

With limited time, let us define the most critical user/client pain points.

  • Intricate organ anatomy is currently interpreted through a series of 2D CT/MRI slices; this is time-consuming and challenging

  • Limited 3D model access despite proven utility

  • VR is shown to be especially beneficial for complex anatomy, however, familiarity skews toward young, white, male users

  • VR’s drawbacks (e.g. motion sickness, increased physical effort)

  • Low adoption of previous VR prototypes at TGH†

  • The more accessible desktop modality has not been considered

  • Efficiency demands from a very busy user base

  • Institution’s prestige at stake

†TGH = Toronto General Hospital, the organization this project serves.

Deriving pain points via affinitizing research findings.

Strategy

Despite a local demand for VR, was it really the best solution?

My clients were already interested in conducting a study comparing the effectiveness of anatomical learning between using 3D models (VR) and using CT scans (desktop). However, I identified a limited pool of research studies that suggested the potential utility of 3D models on desktop displays:

  • For more complex spatial HPB anatomy problems, 2 studies showed that VR > Desktop

  • For simpler spatial HPB anatomy problems, 4 studies showed conflicting findings regarding VR vs desktop

I presented my discoveries and persuaded my clients to incorporate 3D models (desktop) into the comparison, especially given desktop’s relative convenience and familiarity for users. This helped to position my clients to fill an important gap in the literature on surgical education — which aligned with my clients’ core mission — and helped to strategize for the future in terms of choosing the best 3D modality for surgical education.

Methodologies

To generate the above research insights, I used multiple UX research methods.

Research: Client/User Interviews, Field Observations, Competitive Analysis, Literature Review. Sensemaking: Affinity Diagram, Journey Maps, Personas

Define

〰️

Define 〰️

Project Scope

Defining requirements of an MVP that best serves user and client needs: 3 Unity applications.

  • MU-VR: A professional multi-user medical VR platform for collaboratively tackling complex HPB patient anatomy in the context of surgical planning and education

  • RCT-VR and RCT-De: Single-user VR and Desktop counterparts that will be compared in the study (a randomized controlled trial i.e. “RCT”) mentioned in Strategy.

To further refine the scope, I developed the information architecture, user flows, and detailed project requirements, integrating feedback from stakeholders along the way. At this stage, I also developed a detailed timeline that could be continuously adapted.

Crystallizing the scope via information architecture.

Design

〰️

Design 〰️

Proof-of-Concept

How can we communicate VR designs clearly and quickly to cross-functional partners who have hectic schedules? The answer could be a Unity prototype!

This prototype was not only efficient in communicating designs, it also garnered stakeholder confidence and allowed us to audit development resources as early as possible, ensuring smoother progress later on.

VR Onboarding

Research suggests that 52% of consumers have not had their first VR experience, with those who have used it being predominantly young, male, and white. Informal testing with the first prototype reinforced the need for comprehensive and accessible VR onboarding that enhances usability and inclusivity.

To refine the onboarding module and test additional interactions, I made more iterations of the prototypes, gathering user/client insights along the way. In the end, we decided to film and edit a VR device donning video and include in-app VR onboarding modules and onboarding refreshers.

Low- to High-Fidelity

Iterating functional prototypes, despite their efficiency in conveying designs, is expensive — so we used low- to high-fidelity mockups wherever possible.

I produced paper prototypes, storyboards, 3D models, thumbnails, and 2D mockups in order to communicate and refine various aspects of the design.

UI Design

Figma was instrumental in mocking up and finalizing UI.

Brand Identity

It is challenging to secure funding to sustain development in a not-for-profit context for innovative projects unless strong utility and promise are demonstrated. To support this effort, I also established a cohesive and memorable presence via a logo and the name liVeR.

In addition to meeting conventional graphic design requirements, the logo also needed to be respectful towards the extremely sensitive nature of patient anatomy.

Software Development

Through self-learning Unity and scripting fundamentals, seeking the support of my technical advisor, and utilizing ChatGPT, I was able to script the majority of the Unity prototypes and final applications.

Working in a very niche space, my GenAI scripting workflow consisted of ideating with ChatGPT, manually outlining my scripts, filling in the syntax with ChatGPT, and testing and tweaking as needed. Other technologies I used include GitHub, Meta SDKs, and Normcore.

Usability Tests

Usability testing significantly informed and validated my designs. I conducted formal usability tests with 12 end users across the 3 applications.

Key resulting improvements:

  • Optimization of object positions in VR to accommodate broader usage scenarios

  • Incorporation of short, looping onboarding videos that convey instructions accurately and quickly

  • Removal of superfluous features to prevent script disruption

  • Implementation of particular medical terminologies that are familiar to the users

Positive user feedback:

  • Innovative features with high utility

  • Intuitive interactions

  • 9/10 subjective usability rating

  • Perceived seamless integration into current workflow.

Deliver

〰️

Deliver 〰️

Onboarding Module

In addition to designing core interactions themselves to be as intuitive as possible, one of the most critical ways we enhanced inclusivity and usability was via progressively-disclosed, comprehensive, step-wise instructional modules.

Final Product

Through addressing our problem statement and key pain points, the final product received overwhelmingly positive feedback from clients and potential users, and accomplished the following:

MUVR (Canonical Experience Completed, Awaiting Refinements Before Launch)

  • Overcame challenges in securing funding for innovative projects within a non-profit context by showing strong utility and promise

  • Provided an intuitive VR experience for users who are mostly unfamiliar with the modality

  • Addressed underuse of previous prototypes by providing a complete and polished user experience

  • Circumvented VR’s drawbacks (e.g. motion sickness, buggy interactions, unfamiliarity)

  • Integrated the client’s vision of specific features (e.g. cabinets)

RCT-De and RCT-VR (Ready to be Used in Study Once the Appropriate Patient Models are Incorporated)

  • Enhanced long-term strategy for augmenting surgical education

  • Reduced confounding variables in the study by implementing identical UI/user flows and standardizing onboarding

To facilitate smooth project hand-off, the final package included a design document, a figma-based design system, Unity project files, etc.

Reflections

〰️

Reflections 〰️

Key Takeaways

A heartfelt thank you to my clients, advisors, and everyone who supported me along the way. It was an honor and I’m thankful for the lessons.

  1. Having experimented with Midjourney and ChatGPT in this project, I’m so excited to continue exploring how generative AI can enhance workflows.

  2. Being in the weeds of scripting broadened my skills and will enhance my collaboration with engineers and developers in the future.

  3. Successfully collaborating with my stakeholders reinforced the utility of grasping jargon in effective cross-disciplinary communication.

  4. This project was a process of rapid growth. In retrospect, I would implement a design system earlier on to enhance visual consistency.

<

Mobile Website 

Product Designer

>

Mobile AR

Unity Developer, Team Lead