Integrating Change Through Documentation of Experience for Immersive Media

sasha arden (New York)

 

Immersive media (IM) is a class of technologies that aim to create an immersive environment for viewers, such as virtual reality (VR), augmented reality (AR), mixed reality (MR or XR), and 360° or panoramic projection installations. The preservation of immersive media is a growing field of research and practice, whether for historical reference of these technologies, or for re-exhibition of artistic/creative works. IM benefits from existing strategies developed for preservation of complex digital and physical media such as software-based art and installation artworks, but the unique characteristics of IM lead to unique preservation needs. Exploring how these needs may be met through expanded documentation and cataloging practices prompts reflection on how such changes could set larger institutional shifts into motion.

 

Challenges for Conservation of IM

Documentation of experience for IM has been identified as one area for further investigation in Jack McConchie and Tom Ensom’s report on preserving virtual reality artworks [Ensom and McConchie 2021]. The report outlines the complex technologies used in immersive media, and the major concerns for preservation that follow from rapid development of proprietary technologies, commercial access policies, and obsolescence.

Documentation of experience in this context came about from the recognition that experience is an integral part of understanding the significant properties of IM artworks, which could be used to inform how they are reactivated – if possible. That if possible disclaimer is not usually so quickly invoked by conservators in conceptualizing and planning for how to support ongoing lives for artworks. However, in this case there are many potential obstacles in the process of re-exhibiting immersive artworks, and potentially quite a short window of opportunity in which to create effective documentation while their technological ecosystems are functioning.

The various forces exerted over time on such ecosystems, such as technological obsolescence, material degradation, loss of expertise, and changes in socio-technical context[1] create uncertain conditions for supporting immersive media works (figure 1). Hardware might become historical artifacts that break and can’t be repaired or replaced; app development and distribution platforms change, and compatibility with hardware changes; graphics rendering toolchains may fail; and account validation and authentication systems for developers and users will fall out of service. Immersive media uses complex technologies that require expertise to work with effectively, and that expertise is also subject to market forces and other technological and cultural changes over time. Many of these dependencies are completely out of the control of artists, technologists, and preservationists, and there isn’t really a way to re-create the industrial-scale, and sometimes decentralized, infrastructures that support these technologies.

Figure 1: Conservation of immersive media must deal with multiple factors such as technological obsolescence, loss of expertise, material degradation, and changing socio-technical context. There is no guarantee of what the result of conservation efforts over time may be.

Unlike other time-based media, generally speaking, immersive media can’t yet be emulated or migrated effectively. There are efforts to develop open platforms to remove proprietary hardware and software dependencies, such as OpenXR (https://www.khronos.org/openxr/), but that development might well be outpaced by obsolescence of existing artworks. If anything has been learned about conservation of complex technologies, it’s that no two artworks are the same, so there is no cure-all solution to apply across the board. In this scenario, being responsible for preservation necessitates thinking beyond conventional approaches like maintaining physical materials, bit-level file storage, and documenting how the technologies work.

 

The Stakes for Documentation of Experience

With all of these challenges at play, documentation may be the only thing that persists for some immersive media. This potential already raises the stakes for creating high-quality documentation as a bridge to understanding what these artworks were and what they did in the (often not-so-distant) future. It also raises questions in line with preservation of ephemeral mediums like performance: How does the archive of an artwork relate to the artwork itself? And how can future researchers and viewers relate to the artwork through its archive?

One way to create documentation of experience for immersive media as a component of such an archive would be to identify some technological tools and methods to record what happens while an immersive artwork is activated. These techniques, such as screen recording or capturing user inputs and system responses, create a record of interactions that can be viewed by others[2].

But the larger question of involving other people – the “users” who are having the experiences we’re interested in – raises many more questions about the ethics of how and why such information is gathered, and that information’s archival life cycle.

Before going further, I should specify what I mean by experience, especially in terms of creating artwork documentation. To begin, there is no authoritative or ideal Experience to be identified as a reference. This needs to be pointed out in part because the general idea behind documentation – and by extension conventional preservation activities – is to reproduce some referential state of an artwork. That idea is regularly critiqued in present-day discussions of conservation, but reproduction clearly cannot apply to experience! Experience is by nature highly individual, affected by each of our own imperfect bodies that perceive what’s happening, in addition to our life histories and interpretations of events. It’s impossible to have the same experience twice, so how would we model documentation of experience?

When I say experience in the context of artwork documentation, what I mean is: A definitive moment of encounter by a person whose perceptions are contingent and ephemeral, and not reproducible. This experience is influenced by the individual having the encounter, as well as artistic intent, which presumably has some presence and force. Socio-technical context is yet another aspect of experience depending on one’s time and place in culture and technology. Finally, the performance of the technologies being used also contribute to how it is perceived (figure 2). Differences in these ingredients will produce different experiences. While there may well be some overlap between experiences, no two experiences will be identical.

Figure 2: A model of experience in the context of artwork documentation, composed of four major components that are each variable to some degree.

What would we try to preserve of experience, then? What is an achievable goal in this situation? While it’s not possible to make an experience replicable, per se, it is reasonable to make it accessible in some form for the purpose of understanding the nature of encounters with particular immersive media works in a certain socio-technical moment. With these guidelines of what experience means, and what to work toward, it becomes more clear that existing models of documentation practice, access, and use that aspire to reproduce a certain state don’t quite meet these needs. Documentation of experience requires different approaches to be legible and useful in spite of technological and cultural changes over time, and this affords opportunities to revisit why and how documentation is made and used within a collections and conservation setting.

Shifts in Documentation: Emerging Concepts and Practices

One important shift to consider making to documentation practices in order to better integrate experience is to center the user in relation to technology. A user-centered approach facilitates a better understanding of how the relationship between technology and users is conceptualized, and what is prioritized in our documentation methodologies and their results.

Existing documentation models from time-based media[3] largely use what I call an outside-in approach, where material properties determine how content is rendered by technical systems, and thus experienced by viewers or users (figure 3). Information like technical features of hardware and software, along with parameters in the form of specifications and settings, are presumed to reproduce experience through preserving technical behaviors. In this concept, the technological system is the point of origin for the identity of the work and how it is experienced.

Figure 3: An outside-in documentation model, where material properties determine how content is rendered and experienced.

On the other hand, an inside-out approach takes experience as the point of origin, revealing significant properties that guide how content is rendered by an IM technical system (figure 4). Information like motion quality, degree of presence, and reaction times for interactive elements can be used to inform methods of display. This can allow for some flexibility in the technological system, and prioritizes attention to experience while creating documentation.

Figure 4: An inside-out documentation model, where experience reveals significant properties that guide how content is rendered.

Outside-in and inside-out approaches aren’t necessarily opposed to one another. In fact, they work well together, along with artistic intent, to weave a more coherent account of significant properties, and to inform decision-making for documentation and conservation activities over time. Taking an additive, inclusive approach to gathering and using information is aligned with shifts in practice that not only benefit artworks and their audiences, but the cultural sustainability of collecting and exhibiting institutions.

An encouraging shift has started to take hold within the field of conservation around reconsidering how to practice care through our work. There are many influences for this shift, and my thinking is also partly informed by archivists who critique the colonial roots of collection and archiving, and advocate for feminist and queer approaches to the power dynamics inherent to gathering information, keeping it, and granting access to it.[4]

Hélia Marçal’s scholarship on conservation of performance and contemporary art foregrounds the social nature of preservation, focusing on the person-to-person exchanges and first-hand experiences that take place when learning about an artwork [Marçal 2017; Taylor and Marçal 2022]. These social dynamics were tested, iterated, and folded into a methodology for documentation that was developed with a project team comprising members of Tate’s conservation department and visiting scholars [Lawson and Marçal 2022]. A through-line in these citations is the importance of embodied learning, which emphasizes active and present engagement by those people who are enacting the work and/or contributing to its conservation. Implicit in such exchanges are the dependencies on other parties who hold various parts of the work as a whole – knowledge or experience of past iterations, provision of items, services, and other resources required to complete the work, etc. – which the methodology makes explicit as a form of conservation itself.

In a related vein, Annet Dekker has highlighted the “networks of care” that various stakeholder communities form around net art, distributing responsibility and risk across social and geographic areas in ways that defy conventional, institutional preservation practices (which mainly rely on sequestering artworks within collection repositories) [Dekker 2018]. The reality of distributed artworks has itself disrupted collection practices and possibilities for preservation, but the motivations of stakeholders within networks of care also point to the significance of their experiences of the works in taking actions to document and preserve them.

Libby Ireland, Jack McConchie, and Flaminia Fortunato have spoken about their changing practices of relating to artists, highlighting the trust that can be built between people while the institution as an entity presents obstacles such as questionnaire forms and cataloguing systems that don’t accommodate important aspects or conceptual registers of artworks, and rushed exhibition timelines. Ireland and McConchie’s work with the artist Ima-Abasi Okon prompted them to reflect on “[...] collection care’s complicity in the traditional museum drive to own, fix and hold the knowledge around artworks,” and “how our inherited procedures and approaches may be limiting the ability of artworks to live in the museum”[5]. Fortunato shared her experience of collaborative documentation developed along with artist Simnikiwe Buhlungu, which was fostered through meeting outside the institution, only recording informally and by consent after rapport and trust were established, and using analog cassettes, a format central to Buhlungu’s practice[6].

These conservators’ accounts of attention to artists’ needs, what care means to each artist, and a willingness to try new approaches when business-as-usual doesn’t work show that feminist and decolonial ethics can be practiced in creating documentation. Respecting the value of individual perspectives and resisting one-size-fits-all solutions provide supportive grounds for expanding into documentation of experience for artworks.

 

Incorporating Change Through Documentation

How to incorporate experience into documentation and cataloguing processes raises further opportunities to change the status quo of colonial institutional practices and improve the usefulness of institutional records.

Developments in software preservation and game studies have recognized the value of contextualizing assets such as source code for historic software or documentation of game plays [Bensch et al. 2023; de Vos 2018; Lowood 2013; McDonough et al. 2010]. Context in the form of structured metadata might allow a compatible computing environment to be chosen for access. Metadata about computing environments also contextualizes the conditions within which documentation was created, helping to explain the features or behaviors found in such records of past technologies, and establishing metrics of comparison for preservation activities like migration or emulation. Context also enables the social aspect of collected items to be included, which can be especially salient for virtual worlds where communities of game players or users form, or for overtly social platforms/services that involve some level of connection and interaction among their audiences.

Context is important for immersive media experiences as well. As established already, the “ingredients” of an experience such as technologies used, artistic intent, socio-technical factors, and the individual’s interpretations combine to form a particular outcome at a certain moment. Future access to records of immersive media experience will benefit from a systematic way to include such contextual information, both for enabling search within this type of informational framework, and for providing enough material for researchers to stitch together meaning[7].

Immersive media collection items and their documentation could be cataloged with a post-human collections approach as a method for integrating context. As outlined by Fiona Cameron, a post-human collections approach conceptualizes the so-called collection object not simply as an instance of a physical composition, but a composition of interactions between its thingness (its material presence), its historicity, and the values endowed to it [Cameron 2018]. The social and the material come together, forming a different kind of story about why a particular thing is important to people and why it has been preserved.

While Cameron’s scheme expands traditional cataloguing methods, objects themselves may also accrue their own histories through embedded technologies. Steve Benford and his team at the Mixed Reality Laboratory at the University of Nottingham have created an ‘augmented object’ that collects its own documentation through embedded sensors, recording devices, and Artcodes (stylized graphics that function similarly to QR codes) to add to, as well as call on, its digital archive [Tolmie and Benford 2018: 165-181; Benford and Giannachi 2022: 185-196]. The Carolan guitar builds context as it is used, and as users activate these features to make contributions. As its life unfolds, it constitutes its own archive with provenance and maintenance history, rendering an “accountable artefact” which is “an identifiable thing that can provide varying accounts of itself to different audiences in different situations” [Benford and Giannachi 2022: 189].

In Cameron’s expanded approach, the motivations and processes of record making and access also move away from colonial, extractive attitudes and practices, toward working with and for people. Acknowledging the interconnectedness of materiality, events, and their interpretations within meaning is part of a worldview that rejects exclusivity and impositions of authority.

If institutions were to adopt these types of expanded cataloguing practices, or if collected objects themselves come with their documentation in tow, the role of collecting institutions would begin to change. A supporting role for institutions that is grounded primarily in providing infrastructure for open records would shift the dynamic away from gate-keeping, which has been a widely-leveled criticism, and toward broader engagement with their purposes in society. Considering authorship and positionality as part of gathering experiential information acknowledges the power dynamics involved in creating knowledge through valuing others’ perspectives.

Further, the more open framework of contextual information allows collected works to have continued lives and relevance with new audiences over time, rather than foreclosing upon a particular period of significance. Can our documentation practices leave room for others to have their own experiences, in addition to shedding light on those that occurred in the past?[8] Is there some type of encounter enabled by documentation that is not reproductive of what it was, but rather generative in the present moment for understanding how it was or what it does? As a conservator, I am more interested in identifying the conditions for engagement with ongoing lives for artworks and their audiences, and these questions help reveal emerging, but promising, pathways.

 

Conclusion

The ethical framework and values that determine what methods are used to create documentation of experience can affect the quality of information that is gathered, as well as the availability and relevance of the resulting records. That will ultimately affect the possible outcomes for preservation efforts, and for future understanding of experiential works.

More broadly, the approaches and ethics that have been outlined here for immersive media impact upon the entire scope of collecting and preservation efforts through the centering of experience within their purposes. Those efforts involve more than simply recording the right combination of data in a sustainable format. Enabling a continued life for artworks and institutions themselves means moving away from exclusive, proprietary methods of gathering and sharing cultural assets and their documentation, toward inclusive, collaborative methods with a wider scope of participation and interest, while prioritizing the experience of those involved throughout the entire lifecycle.

 

 

Acknowledgements

This article is based on a 2022 research paper written during a part-time internship placement with Tate’s Preserving Immersive Media group, contributing to my degree requirements for New York University’s conservation program in time-based media. Jack McConchie supervised the placement, while Tom Ensom provided crucial guidance and feedback. Support from NYU Conservation Center faculty Christine Frohnert and Hannelore Roemich was much appreciated. Thank you to Annet Dekker and Katrina Sluis for the invitation to share this research as part of the talk, Seeing Double: Documenting mixed reality in the museum and beyond with the Photographer’s Gallery, London (28 February, 2023).

 

Preserving Immersive Media at Tate: https://www.tate.org.uk/about-us/projects/preserving-immersive-media

Research Paper: arden, sasha. “Augmenting Our Approach to Preservation: Documentation of Experience for Immersive Media.”  June 2022. Tate website: https://www.tate.org.uk/documents/1793/Augmenting_Our_Approach_to_Preservation.pdf