Tommy Sharkey | HXI - Human-centered eXtended Intelligence

ARTEMIS: Augmented Reality Technology-Enabled reMote Integrated Surgery

Thu, 23 Sep 2021 00:00:00 +0000

Artistic rendering of ARTEMIS and its features. Left: a Novice Surgeon in Augmented Reality receiving help from a remote expert. Right: a Remote Expert Surgeon in VR interacting with a 3D point-cloud of the patient, and engaging with the novice on a surgical procedure.

Overview

Specialized surgeries are often needed in places where there is no access to surgeons, and timely access to telemedicine helps novice surgeons save lives.

While modern telesurgery is limited to just audio/video, ARTEMIS enables skilled surgeons and novices to share the same virtual space.

In ARTEMIS, expert surgeons in remote sites use Virtual Reality to access a 3D reconstruction of a patient’s body, and instruct novice surgeons on complex procedures as if they were together in the operating room. Novice surgeons in the field can focus on saving the patient’s live while being guided through an intuitive Augmented Reality interface.

More Info here: http://artemis.surgery

Funding and External Collaborations

ARTEMIS is the result of a collaboration between the HXI Lab and the ARC Lab at UC San Diego, the Contextual Roboticsd Institute, and San Diego’s Naval Medical Center.

This project has been funded by the Naval Medical Center San Diego.

Designing and Studying eXtended Reality

Wed, 16 Sep 2026 00:00:00 +0000

Overview

The lab’s applied XR work lives in its own projects: surgical telementoring in ARTEMIS, holographic stroke assessment in HoloStroke, delineation training in Contouring, remote guidance in eXtended Collaboration. This project holds the work where the headset itself is the object of study.

One line concerns XR as a design medium. Immersive interfaces raise questions that flat displays do not: how precisely a person can draw a non-planar curve in mid-air and whether a pen, a controller, or bare hands support that best; where a step of an instruction should sit so it is readable without occluding the object it refers to; how a user interface should be laid out when the interface occupies the room. PintAR lets designers sketch spatial experiences in the space where they will be used, and HoloCPR places resuscitation guidance on the patient. AcuVR takes the same approach to acupuncture training, where students currently learn from 2D atlases and practice on peers: it simulates customized scenarios from medical imaging and standardized anatomy models, including the sensitive areas that make practice on a person risky.

A second line uses XR as a research instrument. A headset reproduces the same situation for every participant, under experimental control, which makes it a measurement tool. A VR paradigm presents alcohol cues in three dimensions and measures attentional bias toward them, which flat pictorial cues cannot elicit. Wheelchair users explore high-fidelity replicas of physical spaces as embodied avatars, assessing accessibility themselves before travelling. Both studies obtain observations that the real setting would make impractical or unsafe.

Collaborations

This work spans collaborations with UC San Diego Anesthesiology and the School of Medicine, the Department of Psychiatry, Adobe Research, UCLA, and Carnegie Mellon University.

eXtended Collaboration

Thu, 23 Sep 2021 00:00:00 +0000

Overview

Mixed Reality has been shown to enhance remote guidance and is especially well-suited for physical tasks where properties of mixed reality can be exploited to offload the effort required for the communication task from the collaborator to the system.

This project enables the design of XR-mediated remote collaboration systems that exploit the strengths of spatial computing. An in-depth understanding of both human communication patterns and the unique affordances of the technology allow us to create new interaction techniques and alternate representations that make it easier for people to collaborate remotely, especially over physical tasks.

Funding and External Collaborations

The eXtended Collaboration project builds on a number of efforts spanning across the ARTEMIS, HoloCPR, and other AR/VR projects, and investigates in-depth the remote collaboration aspect in Mixed Reality.

PrototipAR/PintAR: Rapid Prototyping in Augmented Reality

Mon, 27 Sep 2021 00:00:00 +0000

Overview

Over the last few years, we have seen a growth of developer-oriented tools for Augmented Reality (AR) such as Microsoft’s Mixed Reality toolkit as well as Application Programming Interfaces (APIs) exposed by game engines like Unity3D and Unreal Engine that attempt to ease the entry to AR authoring.

However, these toolkits still require a high degree of programming proficiency, presenting multiple barriers and a steep learning curve for designers or non-programmers, which results in slow iteration cycles that impede creativity and leave designers fatigued.

Low-fidelity prototyping methods, especially sketching, facilitate ideation and experimentation and are natural to users from different backgrounds. In this project we combine the advantages of sketching with the unique affordances of head-mounted AR displays by enabling direct manipulation of digital content in AR while relying on a digital pen user interface for sketching on a tablet. PrototipAR and PintAR allow users to specify interactive behaviors without writing code and provide an implicit state machine to express different stages of an experiences. They also provide users with an aerial overview of digital content to facilitate debugging through a world-in-miniature view.

Overview

This project creates an environment in which students can practice pair programming in an AR environment. Pair programming is a software development technique in which two programmers work together at one workstation, on the same piece of code. Wearing AR headsets, students manipulate code in three-dimensional space, using familiar gestures such as pointing and grabbing. The use of an AR programming environment that immerses students in their code, forms a more intuitive and collaborative computing experience that engages learners with low confidence in programming, while supporting growth in their computational thinking abilities.

This human-centered AR coding platform is being developed for the creation of three-dimensional assets, artwork, and computational logic. This platform will be a merged digital/physical workspace where spatial representations of code, interactive outputs, and user editing activities are simultaneously located. While wearing AR headsets, learners will manipulate virtual code blocks in real space to assemble programs, and they will debug their code by evaluating the representations that they create. The approach will build on the accessibility and sense of play in successful visual learning technologies (e.g., Scratch), but will leverage regular patterns of perception, action, and social interaction in the three-dimensional physical world. The goal is to increase participation and interest in groups traditionally underrepresented in the educational and career pathways of computer science, including females and some minority students, who often exhibit lower confidence in STEM-related abilities relative to other students.

Funding and External Collaborations

The Embodied Coding project is an NSF Education & Human Resource funded project (2017042). It is a collaboration between the HXI Lab and the Arthur C. Clarke Center for Human Imagination.

DIY Masks

Thu, 23 Sep 2021 00:00:00 +0000

Overview

These projects created N95 half masks that were manufactured and donated to hospitals in the San Diego - Tijuana region. The masks' filtration abilities were tested and instructions made for the DIY creation of these masks. Check out the instructions and extra info for detailed testing results, materials, tips, and more.

Silicone Mask

More Info + Instructions

This is a silicone mask. The silicone allows for a much better seal around the face, allows for resuablitiy, replaceable filters, and easy cleaning.

To make the mask, a mold is 3D printed. 2 part silicone (available in popular hardware stores) is poured into the mold and allowed to set. Once set, the silicone is removed, trimmed, and is ready to go. The mold can be re-used an infinite number of times and shared with others.

Triton Mask

More Info + Instructions

This is a polypropelene mask similar to the everyday N95 and surgical masks people wear. It is extremely cheap and easy to manufacture on both individual and massive scales, with a metal nose piece to make the mask form to the face.

To make the mask, cut or laser cut the form from a polypropelene or fabric sheet (dimmensions can be found in More Info). Cut the aluminum strip to size, use a hair straightener, bag sealer, or other hot device to melt the mask edges and nose piece into place. You now have a mask!

UnMapped: Leveraging Experts’ Situated Experiences to Ease Remote Guidance in Collaborative Mixed Reality

Wed, 19 Apr 2023 00:00:00 +0000

ARTEMIS: A Collaborative Mixed-Reality System for Immersive Surgical Telementoring

Sat, 01 May 2021 00:00:00 +0000

CHI2021 Presentation (5 min)

Project

Do You Really Need to Know Where “That” Is? Enhancing Support for Referencing in Collaborative Mixed Reality Environments

Fri, 01 Jan 2021 00:00:00 +0000

Artemis: Mixed-reality environment for immersive surgical telementoring

Wed, 01 Jan 2020 00:00:00 +0000

PintAR: Sketching Spatial Experiences in Augmented Reality

Tue, 01 Jan 2019 00:00:00 +0000

What You Sketch Is What You Get: Quick and Easy Augmented Reality Prototyping with PintAR

Tue, 01 Jan 2019 00:00:00 +0000