Janet Johnson | HXI - Human-centered eXtended Intelligence

CSE 291A / DSC 266R

Thu, 17 Oct 2024 00:00:00 +0000

Human-Centered AI

Background

Artificial intelligence permeates the fabric of human society, and its ability to generate data-driven insights and recommendations has potential benefits at an unprecedented scale, particularly for underserved and marginalized communities. Together with that promise, AI brings new risks: it can deepen existing inequalities, reinforce injustices, and create deeper disconnects within societies. Many of those risks stem from a growing gap between a technology-centric approach to building AI and the complex socio-cultural contexts the technology ends up embedded in.

This course takes a human-first approach instead. A human-first approach means creating AI systems where human perspective and needs drive the technological innovations throughout every stage of a system’s design: data collection, learning models, inference strategies, interaction paradigms, validation, deployment, evaluation, and maintenance.

Course Description

The course teaches future data scientists how to carry that human-centered approach through the decisions they make across the Data Science Pipeline. One question runs through the whole quarter:

How can we use design thinking to ensure data science and machine learning models are more transparent, approachable, and equitable?

Each week focuses on a specific stage of the pipeline, from use case development and the design phase onward, and pairs it with the human-centered topics and tools that belong at that stage. Students work through case studies and engage directly with the socio-technical questions every data scientist should be able to reason about when designing, building, and deploying AI systems.

The course also brings in outside voices on the themes that cut across the whole pipeline, including sessions on ethics, user-centered design, and safety.

Enrollment

The course is offered jointly as CSE 291A (Topics in AI) and DSC 266R, and is open to graduate students across computer science and data science.

CSE 165

Mon, 30 Mar 2026 00:00:00 +0000

3D User Interaction (Spring Quarter)

Background

From video games to mobile augmented reality, 3D interaction is everywhere. Choosing to use 3D input or 3D displays is not by itself enough: 3D user interfaces (3D UIs) have to be designed carefully for the experience to work.

This course covers the full spectrum of emerging applications for 3D UIs and presents an array of pioneering techniques and technologies. It works through the theoretical foundations, analyses devices and techniques, sets out empirically validated design guidelines, and illustrates the key concepts with running case studies.

Classes are held in the XR Lab (CSE 3219), with lectures twice a week and a dedicated lab session on Fridays.

Goals

By the end of the quarter, students will have gained an understanding of:

Essentials of HCI and human factors centered around 3D UIs
How 3D UIs have evolved, and the hard problems that remain
3D UIs in console gaming, VR, augmented reality (AR), mobile and wearable computers, and remote collaboration
How 3D input and output let users perceive and act much as they do in the physical world
3D output devices, including visual, auditory, haptic, and tactile displays
3D input devices, including traditional, special purpose, and direct human input
3D interaction techniques for common tasks such as selection, manipulation, navigation, and system control
Strategies for designing and developing 3D UIs
Evaluating existing 3D UIs, and previewing their future

Course Structure

The course combines team assignments with a final group project. An introductory project gets students working with XR and Unity, followed by three projects built on the material delivered in class, each completed in teams of two using the XR hardware made available for it:

Project 0: Intro to XR and Unity
Project 1: Selection and Manipulation
Project 2: Travel and Wayfinding
Project 3: Embodied Agent and AR

Projects are demonstrated both as a video submission and in person to the course staff. A midterm in Week 7 covers the material up to that point, and the quarter closes with a final group project.

Textbook

Lectures loosely follow LaViola, Kruijff, McMahan, Bowman, and Poupyrev, 3D User Interfaces: Theory and Practice (2nd Edition), Addison-Wesley Professional, 2017. The textbook is recommended rather than required, and an electronic version is available through the UC San Diego library.

Course Description

CSE 165 focuses on the design of three-dimensional user interfaces, VR devices, and interaction techniques. The course consists of lectures, student presentations, and programming assignments, and students create VR applications to learn how VR displays and 3D interactions work.

Prerequisites: CSE 167 or CSE 167R or MATH 155A.

ARTEMIS: Augmented Reality Technology-Enabled reMote Integrated Surgery

Thu, 23 Sep 2021 00:00:00 +0000

Artistic rendering of ARTEMIS and its features. Left: a Novice Surgeon in Augmented Reality receiving help from a remote expert. Right: a Remote Expert Surgeon in VR interacting with a 3D point-cloud of the patient, and engaging with the novice on a surgical procedure.

Overview

Specialized surgeries are often needed in places where there is no access to surgeons, and timely access to telemedicine helps novice surgeons save lives.

While modern telesurgery is limited to just audio/video, ARTEMIS enables skilled surgeons and novices to share the same virtual space.

In ARTEMIS, expert surgeons in remote sites use Virtual Reality to access a 3D reconstruction of a patient’s body, and instruct novice surgeons on complex procedures as if they were together in the operating room. Novice surgeons in the field can focus on saving the patient’s live while being guided through an intuitive Augmented Reality interface.

More Info here: http://artemis.surgery

Funding and External Collaborations

ARTEMIS is the result of a collaboration between the HXI Lab and the ARC Lab at UC San Diego, the Contextual Roboticsd Institute, and San Diego’s Naval Medical Center.

This project has been funded by the Naval Medical Center San Diego.

Designing and Studying eXtended Reality

Wed, 16 Sep 2026 00:00:00 +0000

Overview

The lab’s applied XR work lives in its own projects: surgical telementoring in ARTEMIS, holographic stroke assessment in HoloStroke, delineation training in Contouring, remote guidance in eXtended Collaboration. This project holds the work where the headset itself is the object of study.

One line concerns XR as a design medium. Immersive interfaces raise questions that flat displays do not: how precisely a person can draw a non-planar curve in mid-air and whether a pen, a controller, or bare hands support that best; where a step of an instruction should sit so it is readable without occluding the object it refers to; how a user interface should be laid out when the interface occupies the room. PintAR lets designers sketch spatial experiences in the space where they will be used, and HoloCPR places resuscitation guidance on the patient. AcuVR takes the same approach to acupuncture training, where students currently learn from 2D atlases and practice on peers: it simulates customized scenarios from medical imaging and standardized anatomy models, including the sensitive areas that make practice on a person risky.

A second line uses XR as a research instrument. A headset reproduces the same situation for every participant, under experimental control, which makes it a measurement tool. A VR paradigm presents alcohol cues in three dimensions and measures attentional bias toward them, which flat pictorial cues cannot elicit. Wheelchair users explore high-fidelity replicas of physical spaces as embodied avatars, assessing accessibility themselves before travelling. Both studies obtain observations that the real setting would make impractical or unsafe.

Collaborations

This work spans collaborations with UC San Diego Anesthesiology and the School of Medicine, the Department of Psychiatry, Adobe Research, UCLA, and Carnegie Mellon University.

VOLI: Voice Assistant for Quality of Life and Healthcare Improvement in Aging Populations

Thu, 23 Sep 2021 00:00:00 +0000

Overview

According to the latest US Census Bureau predictions, by 2035 older people are projected to outnumber children for the first time in US history. This brings significant societal challenges based on their unique living and health-related conditions stemming from reduced sensory, motor, and cognitive capabilities, as well as multiple chronic conditions. Technology can play a pivotal role in meeting the needs of older adults in ways that preserve their independence. Voice represents a natural choice for interaction between an aging individual and their caregivers, social networks, and healthcare providers, and it becomes key for those with visual or mobility impairment.

We are working on a personalized and context-aware voice-based digital assistant to improve the quality of life and the healthcare of older adults, and consequently, to reduce caregiving burden and optimize the interactions with healthcare and service providers.

We strive for innovations in natural language understanding, deep learning, and human-computer interfaces that leverage information from EHRs, clinical ontologies, and novel patient-level terminologies to support among others the clinical use case of detecting symptom changes and medication side effects.

More Info here: http://voli.ucsd.edu

Funding and External Collaborations

VOLI is a NIH/NSF Smart and Connected Health (SCH) funded by the National Institute of Aging (NIA) at NIH. It is a collaboration between the HXI Lab and a number of experts at UC San Diego’s Qualcomm Instititute, Computer Science and Engineering, and School of Medicine in healthcare, expert systems in clinical care, EHR integration, the aging population, patient monitoring, patient self-report, machine learning for natural language processing and understanding, experimental prototyping, field studies, and software engineering of large-scale systems.

eXtended Collaboration

Thu, 23 Sep 2021 00:00:00 +0000

Overview

Mixed Reality has been shown to enhance remote guidance and is especially well-suited for physical tasks where properties of mixed reality can be exploited to offload the effort required for the communication task from the collaborator to the system.

This project enables the design of XR-mediated remote collaboration systems that exploit the strengths of spatial computing. An in-depth understanding of both human communication patterns and the unique affordances of the technology allow us to create new interaction techniques and alternate representations that make it easier for people to collaborate remotely, especially over physical tasks.

Funding and External Collaborations

The eXtended Collaboration project builds on a number of efforts spanning across the ARTEMIS, HoloCPR, and other AR/VR projects, and investigates in-depth the remote collaboration aspect in Mixed Reality.

PrototipAR/PintAR: Rapid Prototyping in Augmented Reality

Mon, 27 Sep 2021 00:00:00 +0000

Overview

Over the last few years, we have seen a growth of developer-oriented tools for Augmented Reality (AR) such as Microsoft’s Mixed Reality toolkit as well as Application Programming Interfaces (APIs) exposed by game engines like Unity3D and Unreal Engine that attempt to ease the entry to AR authoring.

However, these toolkits still require a high degree of programming proficiency, presenting multiple barriers and a steep learning curve for designers or non-programmers, which results in slow iteration cycles that impede creativity and leave designers fatigued.

Low-fidelity prototyping methods, especially sketching, facilitate ideation and experimentation and are natural to users from different backgrounds. In this project we combine the advantages of sketching with the unique affordances of head-mounted AR displays by enabling direct manipulation of digital content in AR while relying on a digital pen user interface for sketching on a tablet. PrototipAR and PintAR allow users to specify interactive behaviors without writing code and provide an implicit state machine to express different stages of an experiences. They also provide users with an aerial overview of digital content to facilitate debugging through a world-in-miniature view.