Augmented Reality Note Taking for Museum Professionals

Teammates: Kelly McClinton, Jennifer Wang

My contributions: Team Lead, ideation, concepting, usability testing

This work was published at CHI 2017 in Denver, CO.

Overview

This project focuses on using augmented reality in a novel way, in this case, as a note taking system for museum professionals. We proposed to use a pen as an input device for an augmented reality system that would allow museum professionals to take notes in 3D space, which could then be used and studied by other art historians and museum professionals. Because we did not have access to an AR headset like the Microsoft HoloLens, we had to be creative in testing. We combined several methods, pulling from body storming, participatory design, and standard usability testing. 

Assumptions

At the outset of the project, we simply wanted to explore augmented reality technology, and come up with a novel project in the museum space. We initially focused on museum patrons, but as we continued, we found that museum professionals were a much more interesting group. I was looking at the following questions:

• What interaction methods beyond the standard set are available in AR?
• What can we do in AR that we can't do in other mediums?

Problem

Art historians face a specific set of challenges. They study unique artifacts that are often related to other items they are often not able to access directly, either because they are protected, or on the other side of the world. Because of these limitations, they frequently use photographs to study related objects that are not located in their collection. These photographs can leave out important details that these museum professionals need. In this paper, we examine an Augmented Reality application that would allow users to view these artifacts in three dimensions, and annotate them directly using a stylus pen.

Methodology

• Research on how technology is used in museums
• Personas and interviews to understand our subjects
• Exploration of the museum space to understand how they are structured
• Exploring augmented reality experiences, to understand how things will function in that space

Major Insights

• While museum patrons could make use of augmented reality technology, it would require AR headsets to be ubiquitous, or a museum to invest money in expensive headsets, then build the infrastructure to rent and maintain them. 
• The museum is a quiet space for many people, and AR headset wearers using hand gestures and voice commands in the museum might be distracting. 
• It is very hard for museum professionals to access artifacts they need to study, as they are spread all over the world in museums or private collections, if they exist at all.
• Museum professionals have begun to use 3D models to preserve and study artifacts.

These insights led us away from designing for museum patrons and towards a design for museum professionals.

Early Concepts

Initially, we were designing concepts focused on museum patrons and how we could realize a museum experience in the HoloLens.

Understanding AR

We also wanted to ensure that our  concept would be unique to AR, rather than simply a VR or screen experience forced into AR, so we did an analysis of the various concepts we imagined, trying to determine which could be done in VR or on a screen, and discarding all but those that worked best in AR. We completed this analysis after considering the various features of each technology.

Iterations

Pen-based note taking  

Goal - The design started to coalesce when we realized that current interaction techniques in AR, such as hand gestures and voice commands would not support our user group. They simply weren't precise enough. We needed something tactile and precise, that could work in 3D space. 

Result - We imagined a pen-based interface that would allow users to touch the surface of a statue, that would create a virtual point in the headset for the user to attach to a note, or some other kind of information. 

Pen-based interactions

Goal - Once we hit on the idea of a pen that users would use as an interactive device, we began developing functionality for this device as it related to museum professionals. To do this, we wrote a story involving our persona using our device to study a statue. 

Result - Using the story informed by our research, we were able to understand how a museum professional would approach this artifact, and how an augmented reality system might help them. We solidified how note-taking would work, what types of notes were available, and how they would be shared. We also imagined how buttons on the pen could be used to change modes or activate different features.

These iterations allowed us to move into a testing phase.

Testing

Testing this design presented the following challenges:

• We had no augmented reality headset at the time
• The pen we had designed did not exist in any functional capacity
• We had no access to the priceless artifacts that we hoped to make more accessible.

We needed a testing method that worked like paper prototyping, but in 3D space. 

We performed usability testing on a museum curator, an art history professor, and several art history grad students. In order to test the design, we 3D printed a statue of the Crouching Aphrodite in as large a size as we could with available technology. We then gave our participants a stylus pen from a Microsoft Surface Pro, and gave them several open-ended tasks:

• take a note
• take a measurement
• compare with another statue
• any other tasks they would like to be able to complete

Testing Results

The testing combined bodystorming and participatory design and was incredibly successful in testing the design and then learning more about our user group and how they function as professionals. It also altered the design significantly. We added:

• new context based pen interactions, including using the pen as a level line and a voice recorder.
• additional note taking options, including a hard surface on each side of the statue for the user to hand write notes on

We also learned about how they think about artifacts:

• all testing participants were hesitant to touch the statue in anyway. They revered it, even after we told them it was a plastic model
• multiple types of measurements that museum professionals wanted to make
• additional modular lighting options

Reflection

In this project, I learned how to become a leader on a design team. I have had many leadership roles in the past, but this was the first time I was the most experienced designer, with everyone else on the team looking to me to guide things and make final decisions. I embraced this role, guiding the process for the less experienced designer on the team, and helping the art historian understand design methodologies and processes. 

It was also up to me to decide how to proceed with testing. To do this, I drew on my training using design research methodologies, and combined the methods I needed to get the information required.

As to the questions I wanted to answer, we discovered through our research that hand gestures and voice commands simply weren't precise enough for our user group. This led us to the pen idea fairly quickly. In testing, it worked well, so we only iterated on the functions available and how they would operate. We also worked hard to explore what AR was capable of, and what would work well in AR vs other technologies. While we didn't have a HoloLens at the time, much of our research proved accurate when we acquired a HoloLens soon after this project completed. 

This project also opened my eyes to the vast potential of augmented reality as a new paradigm of computing. I think one of the most interesting things we did in this project was challenge the existing interaction methods that are being studied in AR, namely hand gestures and voice commands. I believe that AR is the future of computing, and that the technology up to this point has been very driven by technology and engineering. It's time for the designers to get involved and start bringing human needs back into this technology.