Mixed-Dimensional Information Spaces

Overview

Information that describes physical work is still authored and consumed in flat documents. A 3D model is reviewed by typing comments into a text box, a repair procedure that plays out across a kitchen arrives as a numbered list on a phone, and a designer using a screen reader has no way to ask what a model looks like. While the task is three-dimensional, its representation is not, and the user absorbs the cost of the translation.

This project builds interfaces that carry content across that boundary, using generative AI and spatial computing. MemoVis creates companion reference images while a reviewer types design feedback, suggesting viewpoints on the 3D model that match the comment and composing an image from them. PaperToPlace transforms existing instruction documents into spatialized mixed-reality experiences, placing each step where it can be read without occluding the object it refers to. SweeperBot opens 3D model repositories to screen reader users, answering natural-language questions about a model by selecting informative viewpoints and describing their content.

Collaborations

This work was carried out largely with Adobe Research, with collaborators including Cuong Nguyen, Jane Hoffswell, Jennifer Healey, Trung Bui, Thibault Groueix, and Alexa Siu.