Where Should We Put It? Layout and Placement Strategies of Documents in Augmented Reality for Collaborative Sensemaking

Introduction Video

https://youtu.be/fCVpxu4tzDw

Full video of Document Layout and Placement Strategies in AR.

We also have a 30-second preview video.

Quick Access

  1. Abstract
  2. Presentation @ ACM CHI ’22
  3. Study Parameters
  4. Tasks
  5. Results and Findings
  6. Design Implications
  7. Initial Study in CHI ’21 Late-Breaking Work
  8. Related Publications

Abstract

Future offices are likely reshaped by Augmented Reality (AR) extending the display space while maintaining awareness of surroundings, and thus promise to support collaborative tasks such as brainstorming or sensemaking.

However, it is unclear how physical surroundings and co-located collaboration influence the spatial organization of virtual content for sensemaking. Therefore, we conducted a study (N=28) to investigate the effect of office environments and work styles during a document classification task using AR with regard to content placement, layout strategies, and sensemaking workflows.

Results show that participants require furniture, especially tables and whiteboards, to assist sensemaking and collaboration regardless of room settings, while generous free spaces (e.g., walls) are likely used when available. Moreover, collaborating participants tend to use furniture despite personal layout preferences. We identified different placement and layout strategies, as well as the transitions in-between. Finally, we propose design implications for future immersive sensemaking applications and beyond.

Presentation @ ACM CHI ’22

The paper will be published in the Proceedings of the ACM CHI 2022 CHI (New Orleans, Louisiana, USA).

It will be presented in-person & online during the conference in May 2021 in the Reasoning and Sensemaking session.

Supplementary Materials

Study Parameters

Physical Surroundings

Physical Surroundings Study Condition
The physical surrounding setting (Fully-furnished vs. Side-furnished) was a between-subjects variable, and a spacious lab room with 8.6 × 5.9m (50.74m2) was selected for the experiment. For the Fully-furnished condition (A), we simulated a common office used for collaboration and workshops in a large, open room with three tables and four office chairs. In addition, we placed multiple whiteboards (one landscape & one portrait analog whiteboard as well as one Microsoft Surface Hub as a digital proxy) and small cabinets in the room to create typical discussion areas.

In the Side-furnished condition (B), we removed the whiteboards, a small cabinet, two tables and two chairs, and placed the remaining two chairs and the table on the side of the room in order to create open spaces. In this way, participants would have higher freedom of movement, which could lead to different types of observable interaction.

Work Styles

Work Styles Study Condition
The work style (Collaborative vs. Individual) was a within-subjects variable, and participants were instructed to perform the same task except for the following difference. For the Individual condition (C), participants were told that it was not necessary to categorize all 50 cards in order to minimize any effect of time pressure, since we wanted to explore the intuitive behavior of the participants for the organization of virtual content. Besides, participants of the individual condition were instructed to think aloud.

In contrast, participants in the Collaborative condition (D) were instructed that they should both agree with the final outcome whilst these teams were encouraged to communicate with each other.

Study Tasks

Task Illustration
For the document classification task, we used two datasets; one with cards of plants and one with cards of animals. Each card consisted of an image, a title, and three phrases describing attributes about the plant or animal, which were designed to encourage sub-categorization and further arrangements.

The plant dataset contained 50 cards of common plants such as different trees, fruits, vegetables, and flowers, as well as textual information about the climate zone of the plant (tropical, moderate, or continental), the pollination method (wind, animal, or other), and the flowering season (spring, summer, or autumn). The animal dataset had similar characteristics. It included 50 cards of well-known animals including both birds and mammals with textual information about the diet (herbivore, carnivore, or omnivore), characteristics (fierce, timid, or smart), and social preference (group, solitary, or domestic) of the animal. The plant images are from Pixabay and various projects on Kaggle including “fruit-and-vegetable-image-recognition”, “flowersrecognition”, “fruits262”, and “fruits299”. The animal images were collected from “animals with attributes 2” and from Pixnio.

We invited 28 participants to perform a typical sensemaking task in AR that consisted of analyzing and organizing digital multi-media documents into structured layouts. Specifically, 50 cards from either the plant or the animal dataset were displayed in stacks evenly spread around the room. Participants were instructed to group the cards into clusters in a way that made sense to them with no specific criteria and that they could rearrange the environment freely if they desired. The classification task took 25 minutes to complete. In the end, the participants were asked to present their final classifications and arrangements to experimenters (5 minutes). Afterward, the semi-structured interview (20 minutes) was conducted.

Results and Findings

General Spatial Layout

General Spatial Layout
Participants mainly used highly spatially structured arrangements for their classifications. This means that in addition to the planar structuring of virtual content (e.g., grid of cards), spatial placement was also considered (e.g., card grids were placed in different room locations). The usage of the physical environment varied: Some participants created geometry-shaped arrangements with less consideration of the furniture and the physical environment while others highly considered it. The above illustrations show the summary of grouped final layouts.

Spatial Arrangement via Separation or Aggregation

Spatial Arrangement Via Separation or Aggregation
Participants were observed to use different spatial arrangement strategies to separate between the clusters as well as to aggregate within the clusters. We distinguish between arrangement strategies for separating grouped cards as categories (i.e., between-cluster grouping) and spatially aggregating grouped cards as subcategories (i.e., within-cluster grouping). The above illustrations show the summary of spatial arrangement strategies via separation or aggregation.

Placement Strategies for Sensemaking

Placement Strategies For Sensemaking
Participants used the available furniture for placement in various manners during the sensemaking activity. Regarding the work styles, in the Individual sessions, there was a more diverse usage of the furniture, i.e., most spots were used for placing clusters or sub-clusters. Moreover, participants tended to be creative in the spatial organization of their categories. However, teams tended to focus on efficiency with regard to the use of their space. Interestingly, participants were more engaged in using furniture for placement in some of the Collaborative sessions. For instance, participants explicitly moved the furniture to adjust the physical surrounding to meet their organizational structuring needs.

Interestingly, we also found that participants linked the real-world experience when working with virtual content. Such a link also leads to a semantic mapping placement, for instance, plants were arranged to attach to the plant of reality (E). Also, the “mole” was placed on the ground (J).

Design Implications

Based on our results, we propose further design implications for guiding the design of future AR-enabled office and the development of AR sensemaking and brainstorming systems.

Future AR-enabled Offices

  • Future AR-enabled offices should have furniture in the environments instead of using purely empty rooms.
  • Future AR-enabled offices should provide plenty of empty surfaces while keeping these surfaces approachable so that intuitive placement in AR can be supported, very much like in reality.
  • We recommend that an AR-enabled office should be a large room, which provides users with plenty of opportunities for content placement, the flexibility of reconfiguration of the physical environment, and the availability of free movement.

AR Sensemaking and Brainstorming Applications

  • Future AR sensemaking and brainstorming applications should offer an automatic layout mechanism.
  • Context-based automatic alignment functions should also be considered.
  • Additional visual assisting functions such as annotating and making visual links should be provided to further enhance the AR sensemaking and brainstorming experience.
  • Future systems should consider physical environments as design opportunities, which aligns with the research trend of situated data analysis.

Initial Study in CHI ’21 Late-Breaking Work

Initial Study in CHI '21 Late-Breaking Work

Abstract

Augmented Reality (AR) has the potential to revolutionize our workspaces, since it considerably extends the limits of current displays while keeping users aware of their collaborators and surroundings.

Collective activities like brainstorming and sensemaking often use space for arranging documents and information and thus will likely benefit from AR-enhanced offices. Until now, there has been very little research on how the physical surroundings might affect virtual content placement for collaborative sensemaking. We therefore conducted an initial study with eight participants in which we compared two different room settings for collaborative image categorization regarding content placement, spatiality, and layout.

We found that participants tend to utilize the room’s vertical surfaces as well as the room’s furniture, particularly through edges and gaps, for placement and organization. We also identified three different spatial layout patterns (panoramic-strip, semi-cylindrical layout, furniture-based distribution) and observed the usage of temporary storage spaces specifically for collaboration.

Presentation @ ACM CHI ’21

The paper has been published and presented in the Proceedings of the 2021 CHI Conference.

For this preliminary study, the pre-print, poster, full video, pre-recorded presentation video, and 30-second preview video are available.

Related Publication