Multimodal Mediation in Translation and Communication of Chinese Museum Culture in the Era of Artificial Intelligence

2.1 Multimodality in Translation and Communication

The reason why multimodal translation and communication have been widely studied by the academic community is that these two disciplines of translation and communication themselves are multimodal in nature. From the perspective of translation studies, the exploration of meaningful transformation between languages has long been centered on the study of language itself, which is, in fact, only one modality of textual information. Meanwhile, “A ‘modality’ is a symbolic resource that is formed in a society and culture to create meaning” (Kress and van Leeuwen 2001:20). However, although verbal signs coexist closely with visual, auditory, tactile, and other non-verbal sign systems, few translation studies have addressed the meaning production and translation transformation of the latter. Back in the mid-twentieth century, Jakobson (1959: 233) first proposed a trichotomy of translation from the perspective of inter-symbolic coding, which includes intra-linguistic translation, inter-linguistic translation, and inter-symbolic translation that “interprets linguistic signs with non-linguistic system signs”. Later, Western translators like Mayoral also recognized that non-linguistic factors such as pictures and music were also involved in the construction of meaning in comics and movies (Mayoral, Kelly, and Gallardo 1988). But at that time, these factors were simply regarded as obstacles in translation, while their communicative functions remained relatively neglected. So multimodal translation was in a relatively slow process of development in the twentieth century despite its initial knowledge.

The further development of multimodal translation is inseparable from the development of communication science. In the 21st century, when globalization is developing rapidly, the popularity of the Internet has popularized the mass communication media underpinned by the definition of communication and signaled the advent of the media age. In a new communication environment characterized by advanced communication tools that have promoted the close and rapid contact of translated information with the public, translation activities are increasingly detached from the single characteristics of paper-based text writing. The content and form of media in discourse transformation and communication have a direct and far-reaching impact on the functionality of symbolic modalities, which brings unprecedented challenges to translation research. Based on the new reality that communication media have become widespread and influential, impacting how translation is perceived and presented, Littau (2011: 261–81) formally considers multimodal signification and translation under the influence of communication media as one of the main contents of translation studies, that is, the “medial turn”. In this theoretical perspective, the differences in media forms drive different modal interventions as an inseparable part of translation activities.

In the era of Artificial Intelligence, museum culture display is a typical multimodal discourse whose information system takes the image as the core resource, and thus various new technologies can be adopted to realize multimodal mediation in the translation and communication of Chinese museum culture. Discussions on multimodal translation have been made by several researchers. Among them is Wu (2021) who has set out to theorize multimodal translation studies in the context of the “medial turn” by giving its history and definition, specifying its types in a systematic manner, mapping its research scope, and introducing current research methodologies concerning translation studies. As for the methodology of Multimodality in Translation and Communication, Tuominen, Jiménez Hurtado, and Ketola (2018) introduce three methods in multimodal translation studies and find that the effect of two modes or multimodes of translation and communication is better than a single mode.

This research, based on the theoretical foundations mentioned above, particularly explores the extent to which multimodality is beneficial in translating and communicating museum culture in China from the perspective of “media turn” multimodal translation studies, aims to investigate. It also aims to identify the ideal model of multimodality and effective methods of multimodal translation and communication to enhance the design of Chinese museum websites and spot exhibitions, with the mediation of Artificial Intelligence.

2.2 Mediation and Translation

In the literature on translation and interpreting, terms such as ‘mediators' (e.g., Archibald and Garzone 2014; Katan 2004; Wang 2017), ‘mediation’ (e.g., Pöchhacker 2008), or ‘translator-mediated’ and/or ‘interpreter-mediated’ communication (e.g., Baker 2006; Tipton 2017, 2021a) are frequently used. However, a close examination of the sources shows that these terms are used by different authors to refer to different phenomena. For example, Archibald and Garzone (2014) use the term to refer to linguistic mediation that aims to “overcome those incompatibilities which stand in the way of transfer of meaning” for the sake of equivalence. Literature on this type of mediation abounds, especially in Translation Studies. In Interpreting Studies, mediation is often discussed in relation to interpreters’ roles (Angelelli 2008) and interpreters’ efforts to resolve (potential) conflicts resulting from the contents of communication (Wang 2017).

There are only a limited number of studies about “multimodal mediation.” For instance, Liang (2021) investigates multilingual and multimodal mediation in online intercultural conversations from a translingual perspective. Similarly, there are few studies about the “mediation of Artificial Intelligence in translation studies.” Therefore, this research aims to explore multimodal and Artificial Intelligence mediation in the translation and communication of Chinese museum culture to fill the research gap to a certain degree.

2.3 Translation and Communication of Chinese Museum Culture in the Era of Artificial Intelligence

The studies on the translation and communication of Chinese museum culture have been primarily conducted by Chinese scholars. Several researchers have investigated Chinese local museums’ translation and communication, such as Wang and Di (2015). Among these, the second and third ones explore the translation and communication effects of a certain local Chinese museum. Lu and Si (2010) explored the application of machine translation to Chinese–English translation of relic texts in museums. There are also a limited number of studies on the translation of Chinese museums from a multimodal perspective, such as Gong and Jia (2015), who conducted a case study of the Forbidden City in Beijing on cultural heritage translation from a multimodal perspective.

In the era of Artificial Intelligence, a number of studies have emerged concerning “Artificial Intelligence and Translation,” especially “CAT Translation” and “Machine Translation,” such as Kwon, Go, and Lee (2020), Cheng and Yang (2021), Jiang and Lu (2021), and Li (2020), etc. Several researchers, such as Shen and Yuan (2018), Tang (2019), Guo et al. (2021), and Wu (2021), have explored the application of new technologies such as AI, mobile technology, big data, etc. in the field of translation. Shen, Othman, and Abdul Aziz (2021) investigated the multimodal discourse analysis of Chinese museum propaganda films from the perspective of new media.

However, few studies have investigated the mediation of Artificial Intelligence to enhance the effect of multimodal translation and communication of Chinese museum culture, especially under the theoretical framework of Multimodality in Translation and Communication. Therefore, it is essential and significant to conduct further and deeper research.

From the previous studies mentioned, it can be seen that although relevant research has emerged concerning the translation and communication of Chinese museum culture, only a small number of studies have investigated the topic combined with the multimodal concept of translation and communication. Additionally, very few scholars have explored multimodal mediation of translation and communication of Chinese museum culture in the era of Artificial Intelligence. Therefore, this research could be the first to apply the theoretical framework of Multimodality in Translation and Communication to find out the ideal model of multimodality, effective multimodal translation and communication methods, as well as the mediation of Artificial Intelligence technologies in enhancing Chinese museums’ website design and spot exhibition in the era of Artificial Intelligence.

4.1 The Mediation of Artificial Intelligence in Translating and Interpreting the Texts of Chinese Museum Culture in Multimodal Ways

Thanks to the rapid development of Artificial Intelligence, the accuracy of machine translation, such as Google Translation, Youdao Translation, and Sogou Translation, has greatly improved. At the present stage, the mediation of Artificial Intelligence in translation can provide good auxiliary reference for translators. People can make adjustments based on machine translation, which can not only improve work efficiency but also ensure the precision and accuracy of translation. We should make full use of emerging technologies and methods such as big data technology, pattern recognition, and algorithm models to understand and study the complex world of translation technology in a more comprehensive, objective, and dynamic way.

Museum text is a type of external publicity text with strong information function. Its most important role is to show the historical information and cultural connotation behind cultural relics to the audience, and it contains many terms (Zhang and Yanhong 2019). Therefore, it is more suitable to rely on the mediation of Artificial Intelligence to improve the efficiency and quality of translation.

4.2 The Mediation of Artificial Intelligence and the Visitor Interaction Experience

Various studies have emerged on the use of Artificial Intelligence in enhancing the museum visitor experience and communication effects. Foreign researchers in education and Artificial Intelligence have shown that machine learning models can predict how long museum visitors will interact with specific exhibits, which has led to the development of new work. In many museums in China, a more direct and efficient solution to enhance visitor interaction experience and increase interest in the visit is through artificially intelligent dialogue bots (chatbots). Chatbots are software applications that can communicate with humans through voice or text to accomplish certain tasks. As their design and functionality continue to improve, they can provide people with interactive and immersive learning experiences in various fields, including education and entertainment.

However, as common Chinese monolingual dialogue robots can only serve Chinese tourists, it is necessary to translate the content of chatbot answers into foreign languages to enable more foreign tourists to learn about Chinese museum culture. By using Artificial Intelligence and multimodal translation together, the dissemination of Chinese museum culture can be promoted comprehensively.

For example, the Qingpu District Museum in Shanghai started building a smart guide system in 2017. With the increase of foreign visitors to the museum, sparked by the success of the first China International Import Expo, an English version of the smart guide system was put into use on October 1, 2020. The English version is integrated with the original Chinese version on the same flatbed guide and contains an in-depth interpretation of 50 key cultural relics at the Qingpu Museum. The system includes not only the traditional translation of the illustrations of the relics themselves but also multimedia video animations of the English interpretation and even human–machine interaction. This improvement enables foreign visitors to communicate with the system’s built-in chatbot and see accurate hand-drawn illustrations of history and thorough storylines, thus bringing to life the originally lifeless, dusty, thousand-year-old cultural relics.

The multimodal format of the illustrations in the picture makes it easier for viewers to understand and appreciate the primitive living conditions of the aborigines in the Qingpu area. Through the illustrations, viewers can see how the aborigines lived by water and grass and how they hunted and gathered for survival materials six thousand years ago.

The intelligent tour guide at the Qingpu District Museum takes a different approach to presenting information. Rather than simply providing information to the audience, it encourages interaction and participation. For instance, when explaining the ornaments or patterns on ancient bronze objects, the audience can click on images shown on the screen to draw patterns by hand. This not only piques their curiosity and interest in learning but also helps them better understand the meaning of the patterns.

At the same time, an animated visual explanation of the assembly process is also a human–computer interaction method which is particularly useful in the introduction of ancient machinery and equipment artifacts. Visitors can click with their hands to see the dynamic diagram of the step-by-step restoration of the ancient production process. That is how the heritage itself can gain a more three-dimensional dynamic understanding. For example:

The picture shows the process of preparing tea in the Song Dynasty. As illustrated, the steps of grinding the tea, sifting powdered tea, measuring tea, filling hot water, and stirring the tea with a whisk are shown in the corresponding animation.

It can be seen that although the dialogues and interactions between Artificial Intelligence and museum visitors are still in their preliminary stages of application, they have already achieved good results, greatly improving the original one-way tour experience and bringing cultural relics to life. With the assistance of multimodal translation that expands the target audience of relevant information dissemination, both Chinese and foreign visitors can experience the beauty of Chinese cultural relics.

4.3 Accurate User Insight and Service with the Mediation of Artificial Intelligence and Big Data

In the era of mobile internet and artificial intelligence, user-oriented thinking is a crucial element. The mediation of big data technology in collecting user demand parameters is indicative of a user-centered service design. The advantage of big data technology is that it can replace causal thinking with relevant thinking and provide insights into the interests and needs of existing users and even potential users based on a large number of calculations and classifications of cultural resource options for user groups (Xu 2021: 159). The mediation of big data technology can help collect the behavioral patterns of users experiencing cultural services. It combines user experience data with cultural resources and user clustering analysis to excavate user experience patterns and demand parameters for user profiling. In this way, it can establish descriptive models that focus on the knowledge structure, behavioral characteristics, and preferred interests of the widest range of users, and then recommend personalized cultural resources in a progressive manner to meet users’ diverse experience needs.

Taking the Longmen Grottoes WeChat service as an example, the WeChat public platform of the Longmen Grottoes is divided into three major functional divisions: “audio explanation,” “ticket booking,” and “parking payment.” Each division has more subdivided services such as “scenic area carrying capacity,” “warm tips,” “Longmen Mall,” and so on. Through the mediation of big data collection and analysis of users in the commonly chosen routes and services, the most concerning functions, efficient and reasonable arrangements are designed to meet the different needs of various groups to the greatest extent. At the same time, the public platform also pushes articles to its followers during traditional festivals such as the Spring Festival, local seasonal events such as “peony blossoms,” as well as relevant cultural hotspots and special events. This ensures that relevant information is accurately conveyed to the target audience while also establishing an interesting and up-to-date image of the Longmen Grottoes in the minds of the spot audience or potential audience.

(The picture above is a screenshot of the conversation page of the Longmen Grottoes public account.)

Given the establishment of a basic user model, the mediation of Artificial Intelligence can promote the continuous adaptation, adjustment, and improvement of accurate services through deep learning. At present, the rapid development of cloud computing and big data technology has laid the foundation of the technical application of deep learning, a milestone in the field of Artificial Intelligence. It also provides the conditions for accurate and rapid analysis of user characteristics. Deep learning can simulate the neural network of the human brain and construct intervals to distinguish the data flow of multi-level nodes, so as to broaden the user classification features of museums’ precise services and shorten the matching time between museums’ cultural resources and users.

Nowadays, the mediation of big data available to blogging organizations under the Artificial Intelligence perspective is diverse in kinds and ample in volume. The big data of users may record the information of users’ behavior, interests, and demand. Meanwhile, institutions can also conduct big data analysis for the cultural resources they own to mine the correlation features between cultural resources and build big data linking relationships of cultural resources through data cleaning, classification, clustering, and mining. With the mediation of various kinds of big data, the convolutional neural network can be used to analyze users’ experience with multi-level data with the help of deep learning technology. By employing advanced techniques such as multimodal and multi-dimensional labels and contextualized, visualized user portraits to delineate users’ distinctive personality traits, a model illuminating the intricate interplay between users, resources, and environment can be formulated. Delving into a comprehensive analysis of customers’ action data permits a granular examination of the factors shaping their decisions, encompassing scene data, service interface dynamics, and decision-making behaviors. This approach unveils critical insights, identifying both deficiencies and peak moments within the user experience. Moreover, through the agile adjustment of user profiles, leveraging insights from recorded user experience characteristics, museums can glean valuable evidence to inform the iterative enhancement of their cultural services.

For example, after launching its WeChat public platform, the Longmen Grottoes cooperates with Tencent Maps to develop the “Smart Tour Longmen” mini App. Relying on the parameter model of user characteristics supported by the big data of audiovisual resources, and combined with the GPS high-speed real-time positioning function under the 5G network, this mini APP can form the correspondence of the “user-resource-context” relationship. To be more specific, based on the user’s real-time location in the scenic spot, it can customize the tour route, recommend the nearest heritage resources and help the user to obtain multimodal audiovisual services such as an intelligent tour guide. Combining the visual and cognitive habits of users and the relevant characteristics of cultural relics, the mini App presents the service interface in the multimodal form of text or voice dialogues with the cartoon character “Ananda” (one of the 10 disciples of Shakyamuni) and lists frequently asked questions to guide the dialogues, such as “I want to go to a certain attraction,” “Where is a certain place,” “The nearest bathroom,” etc.

At the same time, based on further interaction with Ananda, the user’s profile will be personally modified. For example, selecting the question “Where is the Buddha of Lushana” will trigger the option to ask further questions like “Is the Buddha of Lushana the real face of Wu Zetian?” In the process of deepening the user experience, the mediation of big data of user behaviors and preferences becomes more and more abundant. For museums, in generalizing users’ behavioral data and service characteristics, they may broaden the scope and depth of cultural services through multimodal and multi-dimensional portraits of users and develop the most appropriate service capabilities in omnichannel service contexts.

4.4 Flexible Multimodal Communication of Chinese Museum Culture with the Mediation of Digital Forms

Although the large majority of museums have improved their visiting experience with the mediation of technology, in the social context of the COVID-19 pandemic, the opening hours and the number of visitors to museums have been somewhat restricted. Visitors, especially foreign visitors, face many inconveniences when visiting Chinese museums in person. Therefore, it is especially necessary to build online museums with the mediation of Artificial Intelligence. In this field, the Palace Museum and Dunhuang Mogao Grottoes, the two most representative cultural institutions in China, have each made excellent examples of museum cultural communication by digital means in different forms.

In the Palace Museum’s WeChat app “Digital Palace”, visitors can find not only the usual information about visiting the museum, but also a creative entrance to the “Palace Museum Digital Heritage Library”, which exhibits over 83,000 pieces of collections digitally. Both Chinese and foreign visitors can click on a collection to see its high-resolution images and read bilingual classification information and related recommendations. They can also share the collection on other platforms. However, with such a vast collection, visitors may feel overwhelmed. To address this, the digital repository has implemented a mechanism to filter browsing objects by age, classification, and color of the collection.

Moreover, several sections on the app’s home page display cultural relics from different perspectives. For example, the “Daily Palace” column is a calendar that recommends different artifacts by day.

There are also permanent online exhibitions categorized by multimodal forms such as color, zodiac, and motifs, among which the most distinctive one is an interactive online game called “Pocket Palace Craftsman”.

It takes visitors on a mystery tour of the Palace Museum. By moving and clicking the cartoon figure with their eyes and fingers, visitors can tour the manga styled Palace Museum and collect relevant materials to forge iconic Palace objects such as incense burners, stone street lamps and gold pillars. Each level of the game is interspersed with a brief introduction to the relevant elements, so that visitors can subtly feel the culture of the Museum. In essence, with the mediation of multimodality, “Pocket Palace Craftsman” can be considered a “touchable application”. Designed with the limitations of interaction capabilities and devices, such “touchable applications” aim to expand the definition of experience and use interactive touchscreens for aesthetic exploration. “This creates an experience that makes the user’s performance in each moment a creative and unique experience.” (Engberg 2013). With them, the interactive touch screen of smartphones and other devices is used as mediation to provide customized experiences for different visitors with multimodal support, enhancing the exposure and attractiveness of various cultural and museum subjects, and achieving better communication effects.

In another case, “E-Dunhuang” represents another type of online museum (https://www.e-dunhuang.com/). The web-based format demonstrates another successful example of multimodal external communication in the field of culture and museums. With the mediation of technological measures, the Dunhuang Mogao Grottoes and relevant cultural heritage information are digitally collected, processed, and stored, and shared globally through the Internet. “E-Dunhuang” provides a virtual interactive showcase that enables online visitors to go beyond the limits of geography and space, allowing them to explore the classic grottoes and browse the murals and texts of multimodal and multilingual resources in the context of their daily lives. Furthermore, “E-Dunhuang” can reconstruct the main body of artifacts that are missing in reality. Due to their long history, natural and human damage have caused varying degrees of damage to some of the cultural remains of the Dunhuang Mogao Grottoes. However, with the mediation of augmented reality technology, “E-Dunhuang” can apply digital restoration techniques to repair and bridge many parts of the heritage that no longer exist in reality, such as the possible multiple colors and gestures the murals and grottoes may have.

Moreover, it can dynamically restore the complete carving process of the Mogao Grottoes, allowing visitors from around the world to “travel through time and space for thousands of years” to meet their different multimodal and multidimensional needs for sightseeing, appreciation, and research. From the visual modality, “E-Dunhuang” and related promotional videos create symbols of Gansu’s special cultural tourism resources, such as “camel caravans,” “ethnic dancers,” and other humanistic scenes and natural scenes such as the Silk Road and Jiayuguan Pass. Even the designed text can become different multimodal and dynamic visual symbols, which have guidance, interaction, and virtualization. The mediation and combination of various elements constitute a tangible and perceptible Gansu culture with obvious regional characteristics, allowing the audience to better understand the culture and spread it out.

In terms of the auditory modality, “E-Dunhuang” and related promotional films have selected appropriate sound symbols such as character sounds, ground sounds, and scene sounds. The soft and soothing female voice narration can make the audience feel close and integrated, while the soundtrack presents different styles of relaxation and naturalness or seriousness depending on the specific attractions, such as the Tianshui Fuxi Temple with typical ritual music. A comprehensive auditory experience consisting of the multi-dimension of human voice, natural sound, and music can greatly enhance the satisfaction of cultural senses. With the full mediation of Artificial Intelligence and virtualized digital simulation technology, the resulting high-quality and multimodal audio-visual combination builds the communication of overall meaning of museum culture. The visual and auditory modalities complement each other, and can subconsciously bring the audience closer to a deeper experience of perception of the cultural museum subjects and the cultures they represent, strengthening the sense of identity.

On the whole, based on a series of observations of the above-mentioned representative and typical Chinese museums, it has been found that the integration of interdisciplinary technologies such as Artificial Intelligence, online platforms, and new media tools can help cultural and museum institutions produce high-quality and diverse multimodal cultural outcomes, regardless of the specific modalities. The dynamic virtual representations, along with seamless multilingual translations, can infuse inexhaustible life and vitality into Chinese museum cultural subjects and their underlying connotations, thus attracting more visitors and enhancing the effectiveness of multimodal translation and communication of Chinese museum culture. Consequently, these findings provide evidence that “the use of two or more modes of translation and communication is more effective than using a single mode” (Tuominen, Jiménez Hurtado, and Ketola 2018).