Danilo Gasques | HXI - Human-centered eXtended Intelligence

Contouring for Radiation Oncology: Interactive Training and Feedback

Thu, 23 Sep 2021 00:00:00 +0000

Overview

Treating patients with safe and effective radiation therapy depends critically on the precise identification of tumor and nearby normal tissues. This process, known as contouring, refers to identifying and outlining cancer and normal tissues in medical images.

Poor radiation planning has detrimental consequences on patient well-being. Radiation plans that deviate from protocol specifications have substantially decreased survival compared to patients with compliant radiation plans. Given the impact of contouring on patient outcomes, many contouring resources exist. However, practice guidelines rarely translate into real-world clinical practices, primarily due to ineffective methods of development, delivery, and access.

In collaboration with the department of Radiation Medicine at UCSD, we work to replace the traditional one-to-one apprenticeship model with training that is personalized, timely, and available whenever a resident is ready to practice. Three systems carry that agenda.

iContour

iContour is a web-based contouring platform that presents anonymized DICOM cases and gives residents immediate, structured feedback as they delineate. It grew out of needfinding with residents and attendings about when and how contouring feedback actually reaches a trainee, and out of measurements showing that everyday touch devices are accurate enough for real delineation work.

A randomized international trial of iContour has now completed. Secondary analyses of that trial characterize the specific, site-dependent mistakes residents make before and after their clinical rotations, showing that contouring errors are predictable rather than idiosyncratic.

VRContour

VRContour asks what contouring becomes when the anatomy is no longer flat. Residents and attendings work slice by slice on 2D displays even though the structures they are outlining are volumetric. VRContour brings delineation into virtual reality, with sketching techniques designed for 3D exploration and annotation of medical images, and studies which parts of the task actually benefit from immersion.

iConTutor

iConTutor is the next stage. Where iContour delivered practice and feedback, iConTutor aims to turn that into personalized tutoring: using the errors a trainee actually makes, at the disease site they are actually rotating through, to decide what feedback they should see and when.

The completed trial and the mistake analyses are the evidence base. Because errors are predictable and site-dependent, targeted automated tutoring becomes plausible where generic feedback would not be.

Funding and External Collaborations

iContour is funded by Agency for Healthcare Research and Quality (AHRQ). It is a collaboration between the HXI Lab and a number of radiation oncology faculty and residents at UCSD school of medicine.

ARTEMIS: Augmented Reality Technology-Enabled reMote Integrated Surgery

Thu, 23 Sep 2021 00:00:00 +0000

Artistic rendering of ARTEMIS and its features. Left: a Novice Surgeon in Augmented Reality receiving help from a remote expert. Right: a Remote Expert Surgeon in VR interacting with a 3D point-cloud of the patient, and engaging with the novice on a surgical procedure.

Overview

Specialized surgeries are often needed in places where there is no access to surgeons, and timely access to telemedicine helps novice surgeons save lives.

While modern telesurgery is limited to just audio/video, ARTEMIS enables skilled surgeons and novices to share the same virtual space.

In ARTEMIS, expert surgeons in remote sites use Virtual Reality to access a 3D reconstruction of a patient’s body, and instruct novice surgeons on complex procedures as if they were together in the operating room. Novice surgeons in the field can focus on saving the patient’s live while being guided through an intuitive Augmented Reality interface.

More Info here: http://artemis.surgery

Funding and External Collaborations

ARTEMIS is the result of a collaboration between the HXI Lab and the ARC Lab at UC San Diego, the Contextual Roboticsd Institute, and San Diego’s Naval Medical Center.

This project has been funded by the Naval Medical Center San Diego.

Designing and Studying eXtended Reality

Wed, 16 Sep 2026 00:00:00 +0000

Overview

The lab’s applied XR work lives in its own projects: surgical telementoring in ARTEMIS, holographic stroke assessment in HoloStroke, delineation training in Contouring, remote guidance in eXtended Collaboration. This project holds the work where the headset itself is the object of study.

One line concerns XR as a design medium. Immersive interfaces raise questions that flat displays do not: how precisely a person can draw a non-planar curve in mid-air and whether a pen, a controller, or bare hands support that best; where a step of an instruction should sit so it is readable without occluding the object it refers to; how a user interface should be laid out when the interface occupies the room. PintAR lets designers sketch spatial experiences in the space where they will be used, and HoloCPR places resuscitation guidance on the patient. AcuVR takes the same approach to acupuncture training, where students currently learn from 2D atlases and practice on peers: it simulates customized scenarios from medical imaging and standardized anatomy models, including the sensitive areas that make practice on a person risky.

A second line uses XR as a research instrument. A headset reproduces the same situation for every participant, under experimental control, which makes it a measurement tool. A VR paradigm presents alcohol cues in three dimensions and measures attentional bias toward them, which flat pictorial cues cannot elicit. Wheelchair users explore high-fidelity replicas of physical spaces as embodied avatars, assessing accessibility themselves before travelling. Both studies obtain observations that the real setting would make impractical or unsafe.

Collaborations

This work spans collaborations with UC San Diego Anesthesiology and the School of Medicine, the Department of Psychiatry, Adobe Research, UCLA, and Carnegie Mellon University.

eXtended Collaboration

Thu, 23 Sep 2021 00:00:00 +0000

Overview

Mixed Reality has been shown to enhance remote guidance and is especially well-suited for physical tasks where properties of mixed reality can be exploited to offload the effort required for the communication task from the collaborator to the system.

This project enables the design of XR-mediated remote collaboration systems that exploit the strengths of spatial computing. An in-depth understanding of both human communication patterns and the unique affordances of the technology allow us to create new interaction techniques and alternate representations that make it easier for people to collaborate remotely, especially over physical tasks.

Funding and External Collaborations

The eXtended Collaboration project builds on a number of efforts spanning across the ARTEMIS, HoloCPR, and other AR/VR projects, and investigates in-depth the remote collaboration aspect in Mixed Reality.

PrototipAR/PintAR: Rapid Prototyping in Augmented Reality

Mon, 27 Sep 2021 00:00:00 +0000

Overview

Over the last few years, we have seen a growth of developer-oriented tools for Augmented Reality (AR) such as Microsoft’s Mixed Reality toolkit as well as Application Programming Interfaces (APIs) exposed by game engines like Unity3D and Unreal Engine that attempt to ease the entry to AR authoring.

However, these toolkits still require a high degree of programming proficiency, presenting multiple barriers and a steep learning curve for designers or non-programmers, which results in slow iteration cycles that impede creativity and leave designers fatigued.

Low-fidelity prototyping methods, especially sketching, facilitate ideation and experimentation and are natural to users from different backgrounds. In this project we combine the advantages of sketching with the unique affordances of head-mounted AR displays by enabling direct manipulation of digital content in AR while relying on a digital pen user interface for sketching on a tablet. PrototipAR and PintAR allow users to specify interactive behaviors without writing code and provide an implicit state machine to express different stages of an experiences. They also provide users with an aerial overview of digital content to facilitate debugging through a world-in-miniature view.