Introduction

The RETOLD Project (2020-2024), launched by EXARC and funded by Creative Europe, aims to develop a standardised workflow to freely document, digitise and share the heritage of Open-air museums. One of its key features will be the release of a unified digital platform which will improve research capacity and the preservation of digitised heritage data. In addition, it will offer tools that will enable future museum professionals to achieve an active digital presence and enhance participation with communities, thus ensuring the long-term sustainability of these spaces. The main agents involved are: The Universitat Autònoma de Barcelona (UAB), Complexul National Muzeal ASTRA (RO), Steinzeitpark Dithmarschen (DE), Museumsdorf Düppel (DE) with the advice of Nüwa Digital Media Content Production Studios Ltd. (IE) (EXARC, 2021).

Regarding the methodology, the report was commissioned in September 2021 following the development of a benchmark on mobile applications associated with outdoor heritage centres (See Figure 1). Currently, the platform is in the early stages of the strategy phase and the main requirement was to evaluate the current offer on the market. The main goal was to observe how the applications' different information structures responded to the visitors' needs. The report was presented on 6 October 2021, via video-conference with the participation of the different partners.

Approach

The suitability of mobile applications in the interpretation of open-air heritage has been a topic treated and discussed for several years (Grevtsova, 2015). However, it was not until the digital adoption forced by the outbreak of COVID-19 that many museums began to include apps as part of an expanded visitor experience (Özdil, 2020). Ever since, institutions are increasingly opting for the ‘Bring Your Own Device’ (BYOD) philosophy, producing interactive guides accessible online which lead visitors autonomously through the tour.

What benefits does the BYOD philosophy bring? In the first place, the possibility of presenting different levels of explanation with distinctive points of view. Second, the opportunity to offer personalised experiences for different user groups. BYOD also assists institutions in the adherence to health and safety measures, and also in reducing costs (no need to spend more on hardware) and innovating with technologies such as Augmented Reality (AR) (Deakin, 2021; Hardy, 2022). Nonetheless, it is necessary to provide a physical alternative to the BYOD visit that includes those who prefer another type of experience or have little-to-no access to technology (low incomes, low technology literacy, etc). In addition, Open-air Museum apps encounter several obstacles in terms of lack of clarity in the purpose leading to a lack of appreciation of the visit, and also in terms of issues arising from lack of connectivity, such as poor network access, tedious downloads, et cetera. The market is full of stale 'ghost apps' that do not work properly and take too long to load. Therefore, it is always a good idea to think about the project purposes and goals beforehand, considering that in some cases an app might not be the best option.

The first part of the analysis is the ‘User Experience’ (UX) or ‘User-centred Design’ paradigm. Don Norman defined ‘User-centred design’ as "a design philosophy based on the needs and interests of users, with an emphasis on making products understandable and easy to use" (Norman, 2002, p.188). Even though many trends have emerged in recent years that criticise the submission of UX to the demands of corporations and the market (Dunne and Raby, 2013), UX has taken the digital sector by storm, becoming the industry standard. This approach has benefited the research to the extent that it has made it possible to optimise resources by knowing in advance the context and the motivations that guide potential end-users. Paraphrasing Hall (2019, p.3), UX Research1  is a “periscope” (ibid.) that offers a better view of the surroundings.

Study Goal

According to Garrett's scheme (2011, pp.19-24), each digital project is composed by five levels or design phases that are co-dependent on each other: Strategy, Scope, Structure, Skeleton, and Surface (See Figure 2). At the time the study was commissioned, the platform was in the first phase (Strategy), which was designed to explore the market in search of ideas and trends that would help guide the scope. Therefore, the study pursued the goal of identifying and evaluating the ‘usability and information architecture’ of a group of ‘Open-air Museums’ mobile applications. To achieve this, two UX methods were combined to find patterns that provide information and inspire the design of the RETOLD Project platform.

Sample

The selection criteria of the apps that make up the sample followed a hierarchical order. First, EXARC members applications were considered, followed by those from other Open-air Museums not related to archaeology, archaeological museums, and finally a few related to historical monuments or sites. The Google Play Store (Google Ireland Limited, 2022) was used as the main search engine, taking into account different significant criteria such as the developer agent, the total number of downloads, the category, the location, or the user rating.

In the end, the sample was composed of 15 applications2 , 67% of which belonged to archaeological sites (See Figure 3), 40% to EXARC member museums and 50% were developed by the same institution. It should be noted that almost 40% of the total amount of downloads correspond to the application of the Palace of Versailles (Chateâu de Versailles, 2021). When focusing on topics and features, almost 50% are labelled as ‘Travel and Guides’ by the Google Play Store, with an average score of 3.7 out of five, and 15% have ‘Tours’ listed as their main functionality.

Evaluation Methods

In UX Research it is hardly useful to use a single analysis method. The optimum is using triangulation, that is, to combine different data from different sources of information (Salkind, 2010, pp. 1538-1540). Taking into account the requirements and resources of the assignment, it was considered that the methods that best fit are: ‘heuristic evaluation’ and the analysis of ‘information architecture’ (IA) and ‘interaction design’ (ID).

The ‘heuristic evaluation’ is a method which consists of inspecting and evaluating the usability of a website, app, or product (Nielsen, 2020). It evaluates how well a product, website, or app complies with these heuristics to define its usability. Jakob Nielsen (1994a; 1994b) was the first theorist to speak about heuristics and to propose a ‘Heuristic Evaluation Model’. The results are expressed in numerical-based scoring (weighted to the impact on usability) for each heuristic3  until an overall coefficient is obtained (See Figure 4).

The information architecture of the sample was analysed using the ‘sitemap’ or ‘application map’ methodology (See Figure 5). A ‘sitemap’ is a bird's-eye view showing the hierarchy of different pages within an app. It is a diagram that does not have to document every link on every page on the site. In fact, in most cases, that level of detail only confuses and obscures the relevant information. It is crucial to document the conceptual relationships: Which categories go together, and which remain separate? How do the steps in a given interaction sequence fit together? The ‘sitemap' allows the designer to obtain the right blueprints from the analysed apps to map out all the possibilities.

The ‘user journey’ (See Figure 6) is a method widely adopted by the UX community to map the interaction of the end-user with the product and define its interaction design (Salazar, 2021). It is a visualisation method that is also being exported to ​​the museum sector (Moffat and Scott-Songin, 2020) since it provides a holistic view of a particular user in a given scenario. It shows the movement from point A to point B and documents the feelings, thoughts, and obstacles that could be experienced throughout the process. ‘User journeys’ help determine how many screens are needed, what order they should appear, and what components should to be present.

Results

In terms of functionalities4 , most of the appraised apps offer heritage tours which can be accessed from a navigation menu that eases the search. The aid and information features along with the accessibility parameters are also among the most common functionalities (see Figure 7). The most frequent features are the main menu (14 times), tours guides (13 times), and onboarding (10 times). Video/audio-player features are usual (nine times) along with info/help section (nine times) and search functionalities (six times).

Three structure typologies have been found from the information architecture and interaction design analyses. Model A5 , ‘The Guide Model’ (Figure 8) predominates a hierarchical structure of the information combined with a global navigation design. However, in the specific stages the local navigation and sequential architecture take control. A hypothetical user journey could be:

1. The user meets the brand elements and partners.
2. The user navigates and explores using the ‘navigation bar’.
3. The user gets an overall idea of ​​the content and functionalities.
4. The user is introduced to the topic and prepares for the experience.
5. The user starts a tour and chooses one stop.
6. The user plays, stops, reads, and listens/watches the media of the tour.
7. The user finishes the learning experience enjoying the process.

The results of the ‘heuristic evaluation’ (See Table 1) conducted on this first model show an average of seven out of ten in the final computation of all the heuristics. In this particular case, the most prominent heuristic is number six, ‘Recognition Rather Recall’, which means design makes the interface intuitive and keeps the experience simple. Another highlight is number ten, ‘Help and documentation’, which is correct in most cases. In contrast, heuristics one to three still have space for improvement. Especially, 'Visibility of system status' or how appropriate the state of the system is conveyed to users. That means, again, to reduce at the possible minimum the user's frustration and cognitive load. Below, there are heuristics number four, five, and seven. This last one, 'Flexibility and efficiency of use', is far away from the rest. This type of app allows little space for improvisation, creation, or exploration. Their concept is focused on a pre-made journey, so the visitors have to jump from one point to another. The application that stands out the most is that of the Open-air Museum Sagnlandet Lejre (Loby Technologies, 2021).

Table 1. Model A Heurístic Evaluation.

Model B, ‘The Experience-Oriented Model’ (see Figure 9) presents an architecture focused on the delivery of ‘as quickly as possible’ experiences to the user. Because of that, hierarchical and sequential structures and local navigation tend to predominate. A user journey could follow the following sequence:

1. The user meets the brand elements and some basic information.
2. The user sees the menu options and experience specifications.
3. The user chooses an experience when they feel secure and ready.
4. The user lays or interacts with the content and the environment.
5. The user repeats the process until finishing the experience.
6. The user learns and has fun.

Tied with Model A, Model B also obtains a mark of seven in the heuristics (See Table 2). It also follows the trend of Model A and excels in the ‘Recognition rather recall’ area. But it also succeeds in keeping consistency and following industry standards. This model has a reasonably good score in heuristic number three, ‘Match between the system and the real world’. Designing experiences using long-standing conventions ensures that users have a pleasant journey through the interface. The apps that make up this group6  should improve areas such as ‘Visibility of system status’ and ‘Error prevention’ to reduce user’s confusion. It should be noted that in order to offer an interactive experience, for example using Augmented Reality, it is necessary to employ the right indications and make sure to provide ‘exit doors’ to users. More similarities between models A and B can be found by focusing on the lowest score, heuristic number seven or ‘Flexibility, and efficiency of use’. The Experience-Oriented Model tends to have an even more limited user journey and while Model A focuses mainly on the itinerary, Model B puts the interactive experience in the spotlight. The application that stands out above the rest in this model is that of the Dutch historical park De Hondsrug Hunzebosroutes (VR Owl, 2021).

Table 2. Model B Heurístic Evaluation.

Finally, Model C or ‘The Exploration-Oriented Model’ (See Figure 10) has a complex and open architecture, mostly with organic structures and a variety of navigation layouts. The three most common types are: global, supplementary (shortcuts, recommendations), and contextual (links) navigation systems. The user journey in an application with these characteristics could follow the following steps:

1. The user meets the brand elements and customises the experience.
2. The user feels secure and prepared to start.
3. The user glances at all sections of the app at once.
4. The user has immediate access to the main contents of the app.
5. The user searches for specific content (by map, filtering, or scrolling).
6. The user taps on the like icon and keeps the station for a further occasion, acquiring a deeper commitment to the app.
7. The user uses the audio guide playlist to follow the tour.
8. The user interacts with multimedia and feels in absolute control of the experience.
9. The user likes and shares the stop or tour.
10. The user learns and enjoys the process.

This model gets an average score one point higher than the previous two (eight out of ten) and stands out for its excellent use of ‘Consistency and standards’ and its sleek aesthetics and minimalist design (See Table 3). The list of services is diverse and focused on providing information and assistance before the visit. Not every app within this group7  gives an entirely free and open experience, but what characterises this model is the ability to put users in control by allowing them to lead the experience. Contributing to that, there are two main heuristics: ‘User control and freedom’ and ‘Flexibility of use’. The application that stands out by far is the one developed by the Palace of Versailles (Château de Versailles, 2021).

Table 3. Model C Heurístic Evaluation.

Conclusions

Models A, B and C have the same goal: to guide visitors through the tour in the most efficient and enjoyably way possible. Apps under the Exploration-Oriented Model (Model C) present a greater variety in their functionalities, extending their useful life more than the rest. They are also the ones that achieve the best results in the usability analysis. Model C interfaces also present greater consistency across all screens, standardised information, intuitive navigation, and flexibility in use. Nonetheless, this model may not be the most suitable if the project is budget-limited or if long-term maintenance is not planned.

The Guide Model (Model A), characterised by a hierarchical structure and sequential navigation, is by far the most successful type of structure with almost half of the analysed applications belonging to this group. Its entire design is based around the route of the visit, which can be free (for example, following points on a map), or linear, through an audio or video playlist. In addition, it is quite common to combine Augmented Reality interactive experiences with the environment itself, thus exploiting the ‘sense of place’. Its main drawbacks are: 1) The little versatility of its functionalities, that is, they can only be used in one way. 2) These apps are intended to be used only once, which makes many users think that they are not worth downloading8 . 3) Many depend on a good internet connection and the capabilities of smartphones.

The Experience-Oriented Model (Model B), the one with the least representation in the sample, is very similar. As its name suggests, Model B focuses all product decisions on delivering interactive experiences devised for learning with emotion, offering better opportunities to incorporate a ‘storytelling’ approach. Furthermore, it can contribute to breaking the ‘isolation’ of visitors, one of the main criticisms of mobile guides in museums.

Whereas its main feature is gamification, Model B apps are not the only ones incorporating this type of experience; most of the sample merge gaming elements into their user interfaces. Gamified experiences enhance learning by encouraging users to follow the tour itinerary playfully and effortlessly. Some of the most commonly used strategies are quizzes, badges, adventure games, or geolocation games. Despite this, these types of apps are not easy to develop and in most cases of study the design tends to fail, especially in the following areas: 'Visibility of system status', 'Error prevention', 'Flexibility and efficiency of use’ and 'Aesthetic and minimal design'. They also require high programming and maintenance costs.

These three models illustrate the different directions in which the project can be headed. None of them are aimed at a particular type of museum, instead, the conditions of each place must be assessed to find the best possible alternative.

• 1The term ‘research’ referred to when talking about UX Research is not the same as that followed by Social Sciences or Pure Sciences. In UX, an applied and practical use is made of research which is goal-driven and methodologically more flexible.
• 2The 15 apps are: Antwerp Museum App (https://bit.ly/3uf23FH), ArchéoParc Dame Brassempouy (https://bit.ly/3DNAQx3), Cambodunum (https://bit.ly/3LKbQtw), Carnutum App (https://bit.ly/3NSD3Mj), Château de Versailles (https://bit.ly/3r6Nmm2), De Hondsrug Hunzebosroutes (https://bit.ly/3KjJhTd), De Tijdkijker (https://bit.ly/3uWGKIh), Dundas West Open Air Museum (https://bit.ly/3x9rLNW), IndustriekulTOUR Aabach (https://bit.ly/3v3l79n), MUSA Museo Arqueológico (https://bit.ly/3KfQLH2), Museo de Altamira (https://bit.ly/3uYXGxL), Museo de la Evolución Humana (https://bit.ly/3ue5AnG), Parco Colosseo (https://bit.ly/3jd8Yco), Sagnlandet Lejre (https://bit.ly/3DU9Xru), Useeum (https://bit.ly/35KKfJc).
• 3Jakob Nielsen established 10 heuristics: 1. Visibility of system status; 2. Match between system and the real world; 3. User control and freedom; 4. Consistency and standards; 5. Error prevention; 6. Recognition rather than recall; 7. Flexibility and efficiency of use; 8. Aesthetic and minimalist design; 9. Help users recognize, diagnose, and recover from errors; 10. Help and documentation.
• 4In UX, the term ‘functionality’ is used to designate a function or group of functions that are part of what an app offers. For example, to look up for a particular term within an app, you need a ‘search engine’ functionality, which consists of an internal search engine and a form to introduce that term.
• 5Seven apps compose this group: Cambodunum, Carnutum App, Museo de Altamira, ArchéoParc Dame Brassempouy, Museo de la Evolución Humana, De Tijdkijker and Sagnlandet Lejre.
• 6Musa Museo Arqueológico, De Hondsrug Hunzebosroutes and IndustriekulTOUR Aabach.
• 7Antwerp Museum App, Dundas West Open Air Museum, Useeum, Château de Versailles and Parco Colosseo.
• 8Companies such as Smartify (https://smartify.org/) or Google Arts & Culture (https://artsandculture.google.com/) have addressed this issue by creating platforms where different museums and galleries share their routes.