A Boat Is a Boat Is a Boat…Unless It Is a Horse – Rethinking the Role of Typology

1 Introduction

Typology is arguably one of the oldest archaeological tools used even before Montelius formalised the approach in his famous essay “Die Methode” (Montelius, 1903a,b). Since then, this method has had a long history of heated and controversial debate around various issues (Adams & Adams, 1991). It is not the aim to reiterate this history; suffice it to say that typology is messy, biased, and inconsistent, but also one of the best tools in our toolbox. Understandably this led archaeologists to attempt to find ways to make their typologies more consistent and bias free or to abandon them in favour of new means of classifying their materials. Many scholars from Clarke (1978) in the United Kingdom to Cullberg (1968) in Sweden and Sangmeister (1967, 1998) in Germany favoured objective measures, statistics, and computation to achieve this during the processual era of archaeology. Through the critiques of the 1980s and 1990s on the processual view, it became clear that bias is difficult if not impossible to avoid in the archaeological work even if statistical methods are used.

Machine learning and other artificial intelligence methods are now fully established in archaeology (Bickler, 2021), securing that categorisation by the measure of similarity continues to be the focus of archaeological classification. However, too often the focus remains on the freedom from bias that the alghoritms supposedly provide. Hörr and colleagues, for example, theorised that machine learning can save time and remove inconsistencies to make typologies objective and bias free (Hörr, Lindinger, & Brunnett, 2014). However, this ignores that feature selection introduces bias and that there are inherent algorithmic method biases. It may even be said that some form of bias is already created in the various stages of data selection, from excavation and retrieval, reconstruction of broken pottery to archiving. Thus, these methods cannot be bias free. However, they apply the selected parameters consistently, and therefore, their classification is highly consistent.

From this follows that human typology and machine learning classification are both ways to sort data based on the similarity between defined sets of features and characteristics (Fowler, 2017). This is a crucial step back to consider typology solely as a tool to establish similarity that is not automatically conflated with chronology. The links of similarity thus established between the different types need to be interpreted (Stig Sørensen, 2015). In the following, we will concentrate on the difference how alghorithms and humans apply bias in the classification of archaeological data, i.e., machine learning is much more consistent. It will be argued that the interplay between the inconsistency of human classification can be brought into a meaningful dialogue with the consistent classification provided by machine learning to inform how we create typologies for and interpret the outcomes of human creativity such as rock art. That means newer methods do not replace typology and it is important to keep this foundational method in the archaeological toolbox (Stig Sørensen, 2015).

2 Different Data, Different Outcomes

Machine learning methods, like convolutional neural networks, have been used very successfully for LiDAR data, where they helped to speed up workflow and cope with very large datasets through semi-automated detection of similarity in different data such as barrows, Celtic fields, kilns, and much more (Cowley et al., 2020; Traviglia, Cowley, & Lambers, 2016; Trier, Zortea, & Tonning, 2015; Trier, Salberg, & Pilø, 2018; Trier, Cowley, & Waldeland, 2019; Verschoof-van der Vaart & Lambers, 2019; summary in Bickler, 2021). Here, the results have been hailed as a great help to eradicate bias because they create consistent definitions and computer algorithms codify them unambiguously (Davis, 2020). We are not attempting to disparage this important work which will undoubtedly result in significant advancements in archaeological research in the decades to come. However, we agree with Traviglia et al., (2016) that we need to understand how they might improve the work of archaeologists. As such, it is important to understand and remain critical of our data and methodologies. In this sense, it is of note that the data of the examples discussed above already have a high degree of visual consistency, i.e., similarity. Barrows are largely round or oval, and Celtic fields are square. Even if the latter should be delimited by, for example, rivers on several sides and, thus, have a more complex shape, the shape still is consistent because it is defined by the path of the river which follows the topography. This implies that developing unambiguous and consistent definitions requires shapes that are both highly similar internally and highly distinct to other groups. However, the challenge, particularly in archaeological data, is that not all features have consistent forms for any number of reasons; even worse, some data are purposefully ambiguous, such as decorations and art. The rock art of southern Scandinavia is such a complex product of human creativity including all the inconsistencies and idiosyncrasies that make it a difficult material for humans and machines.

3 The Data – Nordic Bronze Age Rock Art

Currently, there are 24,116 registered sites in Scandinavia (Bertilsson, Horn, & Ling, 2021), although this is likely to increase quickly as new sites are discovered continuously. A smaller portion of these panels belong to a tradition of rock art that is distributed in central and northern Scandinavia beginning much earlier (Gjerde, 2010; Lødøden & Mandt, 2010). The rock art images in southern Scandinavian date from the Late Neolithic to the transition to the Iron Age (roughly from 1800–500 BC). However, most of these were made during the Bronze Age between 1800/1700 and 500 BC (Goldhahn & Ling, 2013). We focus on the motifs in southern Sweden, which are the largest pictorial source for the Bronze Age in Europe. These have been applied to the rock through percussive force and depict 500,000+ cupmarks, 20,000+ boats, 6,000+ humans, and thousands of animals, weapons, wagons, and much more. The scene content includes combat, travelling, intercourse, hunting, and farming.

Making rock art was an iterative activity meaning that images, scenes, or panels were not engraved as finished products only to be viewed. There is a long-standing, active, and highly creative tradition to engage with panels that accumulate engravings over time. Images and scenes were updated, changed, and transformed by subsequent generations of rock carvers (Bertilsson, Horn, & Ling, 2021, with previous literature). The rock art in Scandinavia stands out because it has a considerable fluidity of forms. For example, boats were used to construct human features, or humans may resemble boats, animals like deer and horses look like boats turned upside-down, and boats were engraved with animal heads on their stems (Figure 1).

Figure 1

Examples of the ambiguity present in Nordic Bronze Age rock art: (a) boats with animal (horse) heads as prows in Brastad (5:1); (b) a human with a prow-like arm and at the centre an animal-boat hybrid on Aspeberget (Tanum 12:1); (c) a partial horse as comparison for the animal prow-heads in Litsleby (Tanum 72:1); (d) boat with an animal head prow and the left prow superimposes a human-boat hybrid in Vitlycke (Tanum 1:1); and (e) a boat superimposes a human figure so that the stem of the boat also depicts the human’s phallus in Bottna (74:1). All images with the exception of visualizations of 3D models (Horn et al., 2019, 2022). Image (e) is an interpretation after a 3D documentation. Documentation (a) and (d) by Rich Potter and Christian Horn (GU); (b) and (c) by Henrik Zedig (County Administrative Board in West Sweden); and (e) by Ellen Meijer (SHFA).

4 Rock Art and Machine Learning

For machine learning/big data approaches, the Scandinavian rock art is a well-suited case as the Swedish Rock Art Research Archives (SHFA; www.shfa.se) operates with the aim to digitise rock art and all its historical documentation. The accrued data currently consist of 15 TB of documentations of petroglyphs and rock paintings mostly in Sweden, but also in Norway, Denmark, and other countries outside the Nordic sphere. The files reflect over 150 years of research history in rubbings, tracings, drawings, and lately 3D files and visualisations (Bertilsson, 2015).

The project “Rock art in three dimensions” aimed to add a new layer of 3D recordings and to use these data to train a supervised machine learning algorithm (Faster R-CNN) to detect rock carvings in the images semi-automatically. To preserve computing power and time, as well as for the ease of annotation, two visualisation methods were developed for generating single-band and RGB images, which represent depth derived from 3D data as colour values (Horn, Pitman, & Potter, 2019; Horn et al., 2022). This project pioneered using artificial intelligence, specifically machine learning, on the Scandinavian material and the results are encouraging to pursue more sophisticated projects involving more data.

A few other projects have spearheaded new research with machine learning methods on rock art data, for example, from the Alps, Greece, Australia, or China. The first larger-scale attempt was made by the EU-funded “3D-Pitoti” project, especially the PhD thesis Computational analysis of petroglyphs (Seidl, 2016). The aims were to test whether it was possible to determine shapes and pecking styles based on various inputs, i.e., 2D and 3D data. The outcome of the work was that automated classification can conserve time and increase the productivity of rock art researchers. The authors were upfront that this classification was based on human-defined typologies so that bias reduction and consistency were neither aimed for nor a major concern in the work. The same can be said for other publications of the project, which provided a plethora of foundational work (Poier et al., 2016, 2017; Zeppelzauer & Seidl, 2015; Zeppelzauer et al., 2015, 2016).

Kowlessar et al. (2021) attempted to push these approaches further by resolving the style of Australian pictograms from Arnhem Land without prior knowledge of grouping so as to cut out manual typologies to reduce the bias of manual approaches and create consistent chronologies. The data that were classified accepted preconceived styles, which were then confirmed in a one-shot classification based on various transfer learning approaches. The results of this classification were embedded into AlexNet/ImageNet activation space, which distributed them in a mostly straightforward succession. This was then seen as confirmation that the styles of this rock art denote chronological stages, which could be identified automatically without human bias. Tsigkas, Sfikas, Pasialis, Vlachopoulos, and Nikou (2020) have similar aims, attempting to eliminate human classification from the recognition task of rock art sites. They used photographs from three different sites to identify the location of the rock art panel in the image. The methods used were object detection (YOLOv2 and TinyYOLO-v2) and feature pointbased detection (SIFT and RANSAC). The data were annotated beforehand, and in the end, the algorithms could recognise the sites.

All the previously mentioned projects demonstrate important work that benefits the research on rock art and archaeology in general. Still, none eliminates humans from the classification task as long as premade typologies are accepted or the data are annotated for the training process. It is essential to develop machine learning approaches to benefit from (semi-)automation and to develop theory-aware and theoretically sound statistical approaches with interpretable outcomes. The bias that comes with humans is introduced at different points in the pipeline (see also Bickler, 2021). This makes it necessary to consider several questions: Do we need bias-free typologies in archaeology? What is the role of consistency? What can we gain from machine learning when it is applied to the products of human creativity which is inherently subjective and biased? What is the relationship between manual and automated typology?

5 Previous Work and Method

Initially, we trained a Faster R-CNN (Ren, He, Girshick, & Sun, 2016) model on visualisations from 409 laser scans, which were manually annotated. The 3D data were helpful recognising motifs in heavily eroded parts; however, where parts of the bedrock were fractured and blown off, the information is simply lost. The impact of the sample size has been discussed elsewhere (Horn et al., 2022). The algorithm proposed a region of the location of the rock art and assigned a class label. The network was eventually trained on five feature classes and displayed the predictions using bounding boxes. Overall, the mean average precision (mAP) reached 32.5% on the combined classes, with boat performing best at a 64.1% mAP (Horn et al., 2022).

This work tested whether it would be possible to create an infrastructure to automate the keywording process of the SHFA. Continuing from this, the recent work builds on the previous object detection methods but simulates a more human-generated interpretation of the panels by highlighting the rock art rather than outlining with bounding boxes. This links to the discussion of typology, classification, and similarity, because we still rely on human-defined feature classes. However, we aim to enhance typology by highlighting the morphology of rock art features as well as their location and relationship to each other. This works not towards disregarding the human input in such tasks but towards an integrated human–machine interpretation.

6 New Training

Initial models were trained for localisation tasks using semantic segmentation and instance segmentation on 198 visualisations to assess the viability of further training. The semantic segmentation DeepLabv3+ (Chen, Zhu, Papandreou, Schroff, & Adam, 2018) model classifies each pixel and generates a mask, or image, which shows the class assignment for each pixel. Instance segmentation (Mask R-CNN; He, Gkioxari, Dollar, & Girshick, 2017) generates predictions for individual objects, or instances, within a class, rather than considering all pixels of the same class as one object like semantic segmentation. The size of the networks and training datasets were limited by computer hardware and training time, and a trade-off between image quality and image size was considered. Rock art and and an automatically generated background class were included in the dataset. Rock art was manually labelled according to the common motifs in Scandinavian panels: human, cupmark, boat, wagon/plough, animal, other (where there too few examples), foot/shoe, circle, weapon/implement, and lur (horn). As is reflective of the panels in the study area, the cupmark, boat, and human classes were largely represented in the dataset.

With this dataset and training parameters, DeepLabv3+ segmented well-represented classes (Figure 2a), while Mask R-CNN had limited undetected motifs when using a low confidence threshold with manual post-processing or non-maximum suppression (Figure 2b). The accuracy of DeepLabv3+ is helped by the ‘Background’ class, which is also necessary to account for when considering the class balance of the training data. The class included all the background noise like the surface roughness of the rock, ice striations, the image background, etc. The models were tested on a dataset of 15 visualisations, which contained many eroded or difficult to interpret motifs. The foreground accuracy, a measure of the classification similarity between the ground-truth map and model prediction, of the DeepLabv3+ model is 0.84. The mean Intersection over Union, a measure of the overlap between the input labels and the model prediction, is 0.20 for all classes combined, with the best performance in the human and boat classes. The model prediction on new data shows a large portion of correctly classified pixels, with some areas of misclassification. The combined mAP score for the Mask R-CNN is 22.9%, with cupmarks being highest. The downsides to the current Mask R-CNN model are the poor segmentation boundaries, as demonstrated in the prediction on new data, and the potential for overfitting during training. The metrics for both models indicate that further training is necessary to achieve more than a 50% chance that motifs are correctly identified, as well as, demonstrate that these models cannot be used without human verification of the results. It is also important to note the metrics are averaged across the classes and are heavily impacted by the easily classified pixels like those belonging to the background class and those with more representation in the training data. These issues can be corrected as training continues with the addition of new and augmented training data, the adjustment of class labels and image size, and sampling for the correction of class imbalances.

Figure 2

Examples of segmentations created by: (a) DeepLabv3+ with ResNet-50 backbone (pixel-based classification) and (b) Mask R-CNN with ResNet-50 backbone (object-based classification). Classified image visualization of 3D model (Horn et al. 2019, 2022). Documentation by Henrik Zedig (County Administrative Board in West Sweden).

7 AI and Typology

When we consider artificial intelligence, it is worth remembering that even for sophisticated technologies like Facebook’s Deep Face, the amazing aspect is not that it recognises faces – most humans can do the same. The truly fascinating aspect is that a machine without a concept of a human face is able to pick faces out even when they are not the focus of the medium. However, it is this concept that is necessary to understand pieces of art. For example, there are some faces that we recognise as faces which facial recognition has difficulty identifying as such, for example, pointing an iPhone at the painting “Vertumnus (Emperor Rudolph II)” by Giuseppe Arcimboldo which shows the ruler as an agglomeration of fruits and vegetables. On the example of Deep Face, we can also show that artificial intelligence applies learned classifications much more stringent. Photographing several statues shows that some faces are recognised. Human cognition would distinguish these lifeless faces from living human beings. However, facial recognition does not make this distinction because the models were presumably trained on a set of characteristics that were sufficient to identify faces, but did not include the differentiation between living humans and statues. Therefore, the algorithm classifies something as a human face regardless of whether it is biological human or a statue.

Defining such characteristics and looking for a similarity of characteristics across a set of objects or images are similar to the work of archaeologists (Adams & Adams, 1991). Several such characteristics must be similar enough to identify it as a particular object or type, for example to distinguish a dagger from a sword or a flanged axe from a socketed axe. The difference is that trained models are more consistent in their application of the information they have been provided with than any human would be considering the biases inherent in our choices of characteristics, visualisation, and evaluation metrics.

It is important to remember that objectivity does not mean correct; for example, when the iPhone camera algorithm recognises faces, it adjusts the lighting, colour, etc., to enhance human faces. However, the resulting photo may turn out to be worse because the face was in fact a bronze statue, which may have benefitted from different adjustments. It is also possible to theorise that decorations on pottery or designs on jewellery may aim to emulate certain shapes but fail due to a lack of talent of the maker. Where a wave pattern was intended, the final outcome may resemble a zigzag. What then if the maker of the decoration lived in period A during which wave patterns were common, but zigzag patterns only emerged in period C. A consistent classification would come up with the wrong result.

A real-life example of a similar issue is visible in the work of Hörr et al. (2014). In their supervised phase, they reconsidered “incorrectly classified” material and amended their classification. However, their class of biconical vessels unifies vessels that have vastly different volumes (Hörr et al., 2014). It seems doubtful whether they were used in similar ways. Thus, their bias was disregarding volume as a defining characteristic in their typological approach and the model applied this bias consistently. An interpretation of the typological series of the biconical vessels in terms of their functionality will probably come to the wrong conclusion if it assumes that they served the same function because they are the same type.

8 Misclassification, Consistency, and the Merit of Inconsistency

Our models also led to misclassifications, at least from the rock art researcher’s viewpoint. This was the case for the bounding box approach as well as the newer models. In an example from Aspeberget (Tanum 25:1), the model correctly predicted the location and feature class of a larger and a smaller boat (Figure 3). However, somewhat isolated in the space between the stems of the larger boat is another figure. This was given the class label “boat” by the semantic network but would traditionally be identified as an animal in rock art research. Perhaps it was a dog as the short legs, uplifted tail, and short ears would indicate. On the smaller boat, which was correctly identified, the model also predicted the presence of a human, which would also be a misclassification in the traditional view because the network was not trained to recognise the strokes on the boats, which are thought of as indicating crew members, as human. In the scan of a panel in Ekenberg (Östra Eneby 23:1), the model had difficulties predicting the location of a large boat but gave it the correct class. However, the front stem of the boat was misclassified as an animal (Figure 3). While Bronze Age boats may have had carved or otherwise applied animal heads on their constructive elements, in this case, the object itself is part of a boat, and not of an animal. On a panel in Litsleby (Tanum 72:1), the DeepLabv3+ model identified a range of similar images as boats and animals both (Figure 3). Commonly, these images are interpreted as horses ridden by warriors. Thus, the classification as animals would be correct and the label boat would represent a misclassification.

Figure 3

Examples of “misclassifications”: (a) animal in the lower-right centre classified as a boat (Tanum 25:1); (b) large boat with the left stem identified as animal (Östra Eneby 23:1); and (c) DeepLabv3+ classification of horses in Litsleby (Tanum 72:1) partially as animals (orange) and boats (blue). The image was “upside-down” from a human point of view; (d) the images from Litsleby turned 180° to make the image better readable to modern observers. Classified image visualization of 3D model (Horn et al., 2019, 2022). Documentation by Henrik Zedig (County Administrative Board in West Sweden).

Such issues can be amended using data augmentation, which is important in training tasks where features can vary in size and aspect or are partially eroded. For rock art, this is especially crucial when only part of an image is available. In such cases, image rotation can play a key role in the interpretation. This establishes a complex contextual network of similarity on which any classification is based. In this sense, it is not just the defined features and characteristics that establish a type, but their combination. Lacking such a framework makes it difficult to classify images for humans and machines (Figure 4).

Figure 4

Horse or boat? Partial views of a boat on the panel in Bro (Tanum 192:1) showing the horse-headed prow (a) and the aft turned upside down (b). Thus, the context of the image is removed so that it could also be a horse. Visualization of 3D model (Horn et al., 2019, 2022). Documentation by Rich Potter (GU).

Viewers of the images in Ekenberg and Litsleby, especially those acquainted with Nordic Bronze Age rock art, will have no trouble to identify these aforementioned images, for example, the horses, or the boats correctly. Thus, it is easy to dismiss these network-generated classifications as errors. Here, it is important to remember that the network works by identifying visual similarity. So, the misclassification may not be a straightforward error, but a case of a too consistently applied similarity measure. So instead of dismissing these classifications, it may be worth thinking about the reasons why the network identified, for example, a boat instead of a horse. When considering the partial images, it is possible to see that the carvers of rock art used similar constructive elements across a range of figures. For example, the back-hind leg-tail configuration of an animal is similar to how a stem connects to a boat body albeit inverted. The stems sometimes have animal heads, which are obviously similar to how animals are constructed. There is a marked similarity between the overall layout of Early Bronze Age boats and the horns of bull which has been noted earlier (Ling & Rowlands, 2015). Abstract human figures look like halves of boats with changed orientation (Horn, 2018). As a final example, it is possible to point to pairs of shoe sole depictions that when engraved very close to each other, they begin to resemble so-called wheel-crosses. There are many more examples, which enabled the attempt to define styles that were originally seen as diagnostic for different time periods (Almgren, 1987).

We are hardly the first ones to notice the ambiguity of Nordic rock art (Ahlqvist & Vandkilde, 2018; Fahlander, 2020; Goldhahn & Fuglestvedt, 2012; Rédei, Skoglund, & Persson, 2020), nor is it a phenomenon restricted to Scandinavia (Hays-Gilpin, 2004; Tilley, 1991). Engravers did not only use similar constructional elements for convenience. Instead, this means that rock art was created purposefully ambiguous without consistent meaning, at least not in the sense that an either-or-interpretation has to apply. To uncover the full dimensions of such ambiguities, which can be more manifold than the essentially dual examples described above, requires the inconsistent human classification that can recognise that things that look similar may have different meanings or were created purposefully ambiguous. For now, at least, the trained models are at a disadvantage because they have to make a decision classifying the image either in one feature class or the other consistent with the data they were trained with.

9 Creating a Human–Machine Dialogue

The entire deep learning toolset provides exciting new ways to address our data. Generating larger training datasets from new and augmented data, including more training data that are generated by the current models and checked by experts, transfer learning, and refining class balance by adding examples of smaller classes will eventually improve model accuracy, the predictions will be more often “correct,” and it will help to avoid overfitting the models. It is important to remember that all of this is based in some ways on our typologies with which we train the algorithm. This means what the models in the end apply consistently to the data are our own biases and inconsistencies. For this reason, it is important to confront the output with traditional archaeological tools used to identify similarity, because they are naturally biased and in some sense inconsistent (Adams & Adams, 1991).

When it comes to rock art, this will likely show us classifications of motifs that we can identify as something else based on context, i.e., we are inconsistent in our classification of similarity. For example, if a horse head is located on a prow, then the figure should perhaps be better identified as a boat rather than an animal, even though the prow and boat body may be similar to an animal’s body. To recognise this, it is necessary to attempt to understand why the alghortim classified it as one rather than the other. Machine learning makes our biases visible to us by applying them consistently to the rock art data and thereby misclassifying motifs, because the images are as outcomes of human creativity inherently inconsistent. Entering such a “dialogue” with the algorithm can inform us about our biases and the ambiguities of the data at hand. Thus, a hermeneutic loop is established, which will improve our methodologies and interpretations.

To conclude, those working with artificial intelligence should not pretend that we can remove the human from the equation because there is no way that we are able to interpret human-created material without a human view. We should not dismiss apparent “misclassifications” as an error but see them as a way to check and improve our typologies. They provide a chance to learn something about our data. It may even teach us about prehistoric societies themselves, like how Bronze Age carvers used the same constructional elements to make very different, deliberately ambiguous images. Typology and machine learning establish similarities, but our human, biased, inconsistent view informs us that not everything that is similar is the same or has the same meaning. This is why sometimes it is a boat and sometimes a horse, but at other times, it is a horse–boat.