OCR Approaches for Humanities: Applications of Artificial Intelligence/Machine Learning on Transcription and Transliteration of Historical Documents

2.1 Artificial Intelligence and Machine Learning for Natural Language Processing

When it comes to Natural Language Processing (NLP) tasks (e.g., in applications such as digital translation tools, chat bots, voice assistants, etc.), Artificial Neural Networks (ANN) have exceeded the performance of prior machine learning methods. Common types of ANNs include Recurrent Neural Networks (RNN) and Convolutional Neural Networks (CNN). They are heavily inspired by biological nervous systems, in that they are based on “a high number of interconnected computational nodes (referred to as neurons)” that work in “a distributed fashion to collectively learn from the input in order to optimize its final output” (O’Shea and Nash 2015, 1). RNNs are mainly used to detect patterns in sequences of data (Schmidt 2019), whereas CNNs have traditionally been the go-to architecture for pattern recognition within images. CNNs are biased toward preferred solutions during initial training through strong inductive priors (built-in assumptions, design choices, and constraints that leverage the spatial structure of image data to guide a model’s learning process). This avoids weaknesses stemming from a lack of massive sources of ground truth data (such as the scarcity of transcribed early modern archival documents): the training is simplified, and the model learns to better capture the local dependencies in images. Inductive priors include locality (focusing on small image regions using small filters to capture local patterns), translation invariance (recognizing objects regardless of position), hierarchy of features (building complex patterns from simpler ones through multiple convolutional layers), and parameter sharing (applying the same filters across the image to reduce the number of parameters). These inductive priors enable CNNs to be data-efficient, generalize well, and reduce complexity, making them highly effective at executing image processing tasks from limited data.

The landscape of neural network architectures has undergone significant changes in recent years, particularly with the rise of Transformers. Transformers were proposed by Vaswani et al. (2017) as “a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output” that can achieve “significantly more parallelization and can reach a new state of the art in translation quality” (2). They can perform very well on image classification tasks when directly applied to sequences of image patches. When trained on large volumes of data, they can attain excellent results compared to state-of-the-art CNNs. Although initially attention mechanisms were introduced in Long Short-Term Memory (LSTM) networks, enhancing their NLP capabilities, Transformers have rapidly evolved to supplant LSTMs entirely in NLP tasks and are now extending their influence to the realm of computer vision.

As illustrated by Dosovitskiy et al. (2021), Transformers represent a paradigm shift: unlike prior specialized architectures like LSTMs and CNNs, Transformers are designed as highly general computation frameworks. They offer an increased level of flexibility thanks to their dynamic ability to compute connections between inputs on-the-fly. This enhanced general-purpose processing allows Transformers to learn inductive biases from data, potentially discovering more optimal patterns. Given the growing digitization of libraries worldwide, it is now starting to become possible to train Transformer models to decipher the content of scanned historical documents through machine learning-based OCR and perform transcriptions that outperform other approaches based on architectures with built-in or predefined inductive biases.

Most existing OCR tools designed to handle early modern historical documents suffer from common downsides limiting their usability. First and foremost, they require significant end user technical training, even those created with non-technical users in mind, like OCR4all (Reul et al. 2019). Moreover, the goal of a sufficiently accurate, generalized automatic transcription tool remains out of reach: while existing tools showcase promising performance metrics, they are highly specialized for monolingual datasets limited to reduce typeface variability.^[1] Emerging from the field to resolve these issues is READ co-op’s Transkribus online suite of historical document management tools. Transkribus offers a flexible base transcription platform (built on ABBY FineReader) which can be tailored to optimize performance for a particular dataset. End users can select from pre-trained, customized models to run the transcription using one that aligns best with the target document’s features (chronological, linguistic, typographical, etc.). The GUI is user-friendly, and four models have been made available for early modern Spanish texts though Github open access: one trained on 15th–16th-century texts printed in Gothic typefaces (“Spanish Gothic 15th–16th Century”) (Blasut 2022) and three for 17th-century printing conventions: “Spanish Golden Age Prints 1.0,” “Spanish Golden Age Theatre Prints (Spelling Modernization) 1.0,” (Cuéllar 2023) and “Spanish Redonda (Round Script) 16th–17th Century.” (Bazzaco et al. 2022) The first one has reached the highest level of generalization (99.39 % accuracy). The models designed for later 16th- and 17th-century Garamond or “Redonda” style fonts publicize promising CER levels (0.91–3.10), but ultimately, they remain best geared for the specific texts in the training data (e.g., books of chivalry and theatrical plays). Each model struggled with recognizing framed text in Padilla’s Nobleza virtuosa included in the renAIssance project’s training data, suggesting that the Transkribus models have yet to resolve the problem of layout analysis symptomatic of generalized OCR tools.

Layout analysis and line segmentation represent major obstacles to automatic transcription tools regarding historical documents, as datasets are not yet expansive enough to account for layout diversity, material deterioration, or varying scan qualities. For instance, Transkribus can only improve line segmentation generalization by introducing new data to training processes. The renAIssance transcription platform, instead, adopts a flexible segmentation tool into the workflow that allows end-users to manually adjust line bounding boxes per document to avoid and correct errors prior to running the automated transcription. Enabling non-technical users to prepare target documents for more effective layout analysis during pre-processing is a unique contribution of the renAIssance project, which offers the potential to improve prediction outputs while also learning from the new data input to increase the model’s generalized performance (see Section 4.4).

2.2 Inconsistency and Variability in Early Modern Print Sources

Early modern Spanish orthography differs from contemporary spelling conventions. Certain differences can be attributed to the period’s linguistic diversity before languages became standardized to the extent they have today, while in other instances patterns may be predictable with knowledge of archaic orthographical conventions and the material practicalities of moveable type printing. Prescriptive early modern handbooks like Alonso Víctor de Paredes’s 1680 composition manual, Institucion, y origen del arte de la Imprenta, y reglas generales para los componedores are valuable catalogues of orthographic precepts one can expect to find in printed texts from the period, especially when calibrated by informed expectations about the abundance of examples in the real-world archive that diverge from those guidelines. This background knowledge is invaluable when training software models to identify characters and shorthand techniques no longer typical to current usage, and to reliably convert and transliterate them for a modernized output text (Figure 1).

Figure 1:

Depiction of the type sorts found in an early modern compositor’s case, from Paredes’s Institucion, y origen del arte de la Imprenta … , fol. 9r.

For instance, archival sources often interchangeably render ‘u’ and ‘v’ letters, and there are also multiple characters used for the lowercase letter ‘s.’ In practice, orthographical consistency is further mitigated in early modern print documents by any number of pragmatic factors a compositor could encounter in preparing a block for press printing, including the breakage of type letters or running out of a particular letter, among others. Moreover, letter substitutions were often not carried out predictably or consistently, commonly displaying multiple spellings of a word within a single text or even the same sequence of letters rendered by different typeset characters. The inconsistency is due to a lack of distinction between certain letters from antiquity (e.g., ‘u’ and ‘v’), until the seventeenth century brought a shift towards modern distinctions such as ‘v’ becoming a consonant and ‘u’ a vowel. This variability must be accounted for in the design phase of any AI/ML algorithm used for OCR. renAIssance opted for a different approach than Transkribus and Google Cloud Vision; instead of fine-tuning a base platform like ABBY FineReader for a specific dataset or attempting a fully generalized model, renAIssance initially focused on a specific task: early modern framed prose (see Figure 4), which generally enjoys a uniform page layout that eases the burden on algorithms when analyzing what parts of an image consist of text. Coding awareness of lines dividing the main corpus, marginalia, and the outer bounds (i.e., layout analysis) allowed renAIssance to quickly train the model to differentiate these elements.

3.1 An Expanding Dataset: Growing the Pool of Archival and Historical Documents

The rationale for selecting the initially narrow dataset based on Padilla’s Nobleza Virtuosa was to minimize expected printing inconsistencies, as her work is representative of texts with a framed prose layout that have never been transcribed before. It is available in high quality digital scan format from the Biblioteca Digital Hispánica supported by the National Library of Spain (with subsequent tomes also accessible from the digital repository Hathitrust). These scans allow researchers to access copies in near pristine condition, with well-preserved pages that exhibit minimal ink bleed-through, deterioration, or other textual noise such as handwritten annotations, all of which helped facilitate identifying each ML algorithm’s viability early on. The dataset would be progressively expanded when it became necessary due to diminishing returns in training the algorithm (i.e., the code approached a performance ceiling). To accelerate this process, marginalia present in the original sources were ignored in favor of focusing on the main text, and Spanish accentuation was also overlooked due to early modern printing practices not aligning with contemporary use of accents (thus increasing ML complexity with no tangible benefit to the final transcription). As the four main approaches began to take shape, the dataset was broadened to include another 32-page extract from Padilla’s Noble perfecto y segunda parte de la Nobleza virtuosa (1639), which was printed following the same framed prose layout as its precursor, to better evaluate the models’ performance.

The dataset was progressively enlarged in two additional phases: the first involved 27 archival sources gathered at the Spanish National Library in Madrid. These texts belong to the PORCONES classification: judicial documents printed by Spanish governmental offices detailing criminal proceedings.^[2] Published between 1621 and 1693, these archival sources used letters and variations in printing practices that resembled – but were not identical to – those from Padilla’s texts. These additions increased the source pool with 363 folios, 9 of which were manually transcribed to provide ground truth for the ML models. The new documents’ relative consistency with the initial materials facilitated further training of the algorithms: some were framed prose while a few were unframed, providing the opportunity to progressively challenge the algorithms (i.e., to test layout analysis on the frameless text) while still maintaining a controlled training environment.

The second phase infused greater complexity via 10 sources that displayed far greater variability in letter rendering, typefaces, and printing practices. The additional corpus spread chronologically to include materials from the mid-sixteenth to the mid-eighteenth centuries: Luís Milán’s Libro intitulado el Cortesano (1561), the extensive proceedings of a 1590 criminal case, Juan Benito Guardiola’s Tratado de la nobleza (1591), the Constituciones Sinodales de Calahorra (1602), Sebastián de Covarrubias’s Tesoro de la lengua castellana o española (1611), the Recopilacion de las leyes destos reynos (1640), Andrés Mendo’s Príncipe perfecto y ministros aiustados (1662), Paredes’s aforementioned Institucion, y origen del arte de la Imprenta … (1680), Antonio de Ezcaray’s Vozes del dolor nacidas de la multitud de pecados (1691), and Fausto Agustín de Buendía’s Instrucción de christiana y política cortesanía con Dios y con hombres (1740). This final expansion represented a sizable training dataset increase for a total corpus consisting of 931 pages, 2082 folios, and 61 manual transcriptions as ground truth references to train the code. At this stage, printing styles increased in variety, forcing the algorithms to adapt when processing scanned pages. While this represents a growing generalization of the AI/ML model, it remained primarily trained on Renaissance and early Baroque sources.

To resolve the expected issues of variability in the final dataset, four of the more common printing features of this nature (detailed below) were targeted for the ML models to be able to detect and resolve by altering the output text to follow modernized spelling conventions. All four features were addressed throughout this study, from model training to fine tuning and problem solving. Inevitably, all four assumptions are inherently imperfect: they cannot yield consistently flawless orthographical outputs because, by design, setting parameters to simplify the immense potential variability in early modern texts is an impossible task. Nevertheless, the aim was to minimize spelling errors, which would result in a corresponding reduction in the human effort required to manually correct remaining errors in the output transcription. The researchers acknowledge that these assumptions do not currently fit all purposes: setting the parameters for modernizing the text was motivated by the code-training benefits that result from reducing variability. While this approach was most effective for renAIssance’s goals, efforts to refine this code are ongoing. Once the algorithms reach an even higher degree of accuracy, such modernizations and simplifications may be removed to allow the more capable AI/ML model to generate both diplomatic and modernized transcriptions depending on user needs. This limitation for the present study, as well as strategic suggestions for future research, are examined further in the discussion section.

3.2 Machine Learning Approaches

The renAIssance opted for four state-of-the-art machine-learning models for the transcription of historical archival documents that do not follow standard printing practices. Each of the four approaches has strengths and weaknesses. A brief description of each of the four methods employed follows.

4.1 A Convolutional Recurrent Neural Network with a CRAFT Addition

The first approach is based on a recurrent CNN architecture (CRNN), as introduced by Shi et al. (2017), which trained on the dataset after having the source PDFs pre-converted into images for processing. By applying Baek et al.’s (2019) well-regarded Character Region Awareness for Text Detection model (CRAFT), this approach facilitated the precise detection of bounding boxes around text regions, crucial for generating ground truth labels and characterizing text instances effectively. Bounding box coordinates were split and sorted based on vertical proximity and horizontal alignment to ensure that words within sentences were logically sequenced, adhering to the natural reading order of Spanish text. Input data was organized for the CRNN model by associating images with corresponding word names and defining acceptable levels of Connectionist Temporal Classification (CTC) loss function. CTC is a process that sums all possible alignments of the input sequence (characters in the text) that could result in the target output (transcription), considering all valid ways to map the input sequence to the output sequence. The loss value is differentiable with respect to each node, therefore, the closest the loss value is to 0, the more accurate the algorithm is, thus learning to correctly predict the expected letters. The CRNN approach produced a CTC loss that approximated 1 after pre-training and fine-tuning, but further data augmentation was required to increase its accuracy (Figure 2).

Figure 2:

Example of completed data augmentation involving +/− 5-degree rotation as well as insertion of randomized noise via Gaussian filter.

Given that the CRNN model was trained on the target dataset alone, without NLP tools pre-trained on existing datasets, data augmentation processes were necessary to drastically increase the training data pool. This process added complexity to the task, challenging the in-training code to identify similar, but not identical looking text, and thus improved its prediction capabilities. Data augmentation led to a decrease in CTC loss to 0.1 in training as well as after testing for 15 epochs, which was also aided by a learning rate scheduler set to 1e−3. The function of the scheduler automatically decreases the model’s learning rate as the number of epochs increases until there is a negligible change in CTC loss, and thus in any noticeable code improvement. By reducing the learning rate by a factor of 0.5 every 3 epochs (with a minimum rate of 1e−6, every 6 epochs), difficulty is gradually increased to keep challenging the model to become more accurate. A major obstacle for this approach was handling typeface variability, which was solved by re-training the code with the complete renAIssance dataset once it was fully expanded. Figure 3 shows the capability of this CRNN approach after training to transcribe text from a page it has never seen before: accuracy, although not yet perfect given that the model is still considered “in-progress”, is remarkably high with minor errors in “ooligaciones” (failing to properly identify the ‘b’) and “qunie” (instead of ‘quie + n’) and an unexpected result in “ojos” instead of the simple ‘a’ in the source.^[3] This approach achieved a CER of 0.0282, which equates to an accuracy that rose to 95.03 % at the character level. Like the SeqCLR approach, this model outputs single word data, rendering WER values incalculable for accuracy.

Figure 3:

Example of CRNN model performance on previously unprocessed text, after training was completed through data augmentation.

4.2 SeqCLR and the Limitations of Contrastive Learning

The second approach consists of a deep learning model using self-supervised learning. Instead of using SeqCLR for word level text recognition as proposed by Liu et al. (2022), the renAIssance team used it for line-level recognition by optimizing the encoder’s output (i.e., frame sequences), integrating Baek et al.’s (2019) CRAFT model and PyTorch (a framework for building deep learning models) to construct a pipeline that could deliver greater accuracy in word-level recognition (Baek et al. 2019). Our hypothesis was that a cleaner source frame would make identification easier in later stages. This architecture combines ResNet50 (a popular architecture used for computer vision tasks) and a 2-layer Bidirectional LSTM as the encoder (capable of bidirectional sequential processing both in the order of occurrence and in reverse), as well as an Attention LSTM Decoder. The ML model was trained on the archival sources, and automatically generated a dataset containing up to 700,000 synthetic word images with annotated bounding boxes, which allowed the model to focus on the main text instead of the marginalia.^[4] Essential features from the data were extracted by contrasting text images with their transformed copies, including vertical crop, Gaussian blur, random perspective, and random affine transformations. The decoder was trained using both the automatically generated images as well as the 3966 archival folio page renAIssance dataset. CER was used to evaluate the model’s recognition accuracy, as it is generally deemed the most appropriate metric in the field for a decoder predicting characters (Figure 4).

Figure 4:

Example of annotated bounding boxes as used to train the model.

The initial learning rate was set to 0.01 for the CL (3 epochs) and fine-tuning phases (10 epochs). The loss, which indicates how well the model learns during training, remained constant, implying that the model did not learn effectively. In fact, SeqCLR performed better without CL than with it, contrary to the results reported in Liu et al. (2022): with CL, CER exceeded 10 %, whereas without CL, CER decreased to just 0.1. Hypothesizing that implementing CL compromised SeqCLR’s performance forcing the model to learn unrelated information, the team altered two essential values: first, the temperature of the CL loss function, a hyperparameter that determines insensitivity to differences between sequences. Given that the features the model encounters during CL are abstract and difficult to distinguish, temperature was lowered to 0.5, which gradually decreased the loss value during this phase. Second, the team decreased the learning rate during fine-tuning, which dictates how much the model’s parameters are adjusted during training, to 0.0001. These changes brought CL to a 1 % improvement in CER over an approach without CL after 30 fine-tuning epochs, signaling that CL is a valuable tool that can eliminate the need for synthetic data. In parallel with the SeqCLR method, the team explored the potential of PerSec to further increase accuracy with CL. Unlike SeqCLR, PerSec repeatedly demonstrated a gradually decreasing CER during the CL phase in Liu et al. (2022). However, the fine-tuning phase generated poor results, producing identical outputs regardless of input. Further research is required to overcome this deficiency; as the CL components Stroke Context Perceiver and Semantic Context Perceiver may be overfitted (when an algorithm fits too closely to its training data and thus can’t make accurate predictions) to the task, reducing generalization.^[5] Despite these difficulties, this approach still managed to achieve a CER of 0.041, representing a level of accuracy of 95.81 % at the character level. Given that this model uses CRAFT to output data as segmented single words rather than complete sentence lines, WER values do not apply as a calculable metric for accuracy.

4.3 Vision Transformers

The third approach focuses on creating a ViT-based model. Transformers excel at handling sets of tokens through the mechanism of attention, involving a massive computational load as the number of tokens increases. Their application to images, however, can be challenging given that they must process rasterizations of pixels containing enormous numbers of individual data points. A simple 250 × 250 pixel image, while exceedingly small for most contemporary commercial uses in 2024, represents an oversized computational load for a Transformer: every pixel must attend to every other pixel in the training phase, leading to ((250)²)² or 3,906,250,000 connections; billions of calculations that make such an arithmetical processing impractical for OCR. Instead of performing attention over individual pixels, the ViT operates in image patches such as 16 × 16 pixels, significantly reducing the computational burden (Dosovitskiy et al. 2021). The renAIssance dataset scans were first processed through morphological dilation and binarization to remove visual artifacts through masking, as seen in Figure 5. The images were then divided and fed to the model, unrolled into vectors, and processed with the Transformer along with embeddings that were encoded considering position and dimension using sine and cosine functions. In this approach to OCR, positional embeddings are crucial for a successful model, given that transformers are inherently permutation invariant: changing the order of the input tokens would not affect the resulting output, so the embeddings inform the model about the position of each patch within the original image.

Figure 5:

Example of data processing through morphological dilation, binarization, and segmentation to achieve an artifact-free image for use in model training.

This model was trained on both non-augmented and augmented datasets with a cross-entropy loss function, which is useful to measure difference between the predicted probability distribution for each of the tokens in a sequence. However, it evaluates each token independently, which can lead to suboptimal sequence generation, as it doesn’t account for the overall sequence quality. The approach started displaying signs of exposure bias, which, given that it trained on ground truth sequences while it generated its own predictions, can sometimes lead to discrepancies between training and inference performance. To solve this, the team shifted to use a beam search loss, which is often used to generate sequences because it explores multiple potential output sequences and selects the best one based on a scoring mechanism. By evaluating multiple candidate sequences and their likelihoods, the beam search loss optimizes for the best sequences by considering the sequence as a whole, improving coherence and accuracy in text generation. This change mitigated exposure bias and provided a more robust training signal for the model. These refinements achieved a CER or 0.049 and WER of 0.099, indicating a high level of accuracy at 95 % for characters and 90 % at the word level.^[6]

4.4 Enhancing CNNs with TrOCR

The fourth and final approach began as a comparison between the MaskOCR and TrOCR methods applied to the renAIssance dataset. However, it soon became evident that the latter was a better fit for early modern archival sources, because (1) TrOCR was better optimized for text-to-sequence conversion, (2) it leveraged a pretrained language model that enhanced linguistic structure and context handling, and (3) it proved easier to fine-tune than MaskOCR. For these reasons and given TrOCR’s generalizability as well as its robust initial renderings of seventeenth-century Spanish documents, MaskOCR was de-emphasized. The team expected that TrOCR applied to a CNN should surpass a traditional CNN-based approach, given that the latter might face limitations in handling diverse and complex dataset variability, such as the common printing variances encountered in early modern sources.

Data was pre-processed by converting raw scanned text images into formats that are optimal for model inference: this workflow involved a variety of stages that included PDF preprocessing, rescaling, binarization, noise removal, dilation/erosion, rotation/de-skewing, borders, transparency/alpha channel, and line segmentation. This approach also integrated Baek et al.’s (2019) CRAFT model to segment pages into individual lines to execute preprocessing and increase the chances of optimal detection performance. With text lines detected and segmented, individual line segments were matched with their corresponding source text to create a comprehensive dataset for training the OCR model (see Figure 6). Line segmentation was improved by using OpenCV-based methods to better understand the structure of printed texts and remove outliers. Finally, the TrOCR model was fine-tuned on Spanish text by resizing images and dividing them into 16 × 16 patches as described on the ViT study carried out by Dosovitskiy et al. (2021). By establishing a correct ground truth with the provided transcriptions, the code was trained to account for printing practice variability in order to learn contextual representation at the character level, which was expected to aid in resolving ambiguities such as the interchangeability of ‘u’/’v’ as well as the characters ‘f’/‘ſ’/‘s’.

Figure 6:

Processing steps from a raw scanned image, to deskewing, binarization, noise removal, border removal, and final line segmentation filtered for padding, thresholds, overlapping texts, marginalia, headers, and footers.

The model was trained to account for printing practice variability within the initial dataset. This helped the algorithm to learn contextual representation at the character level, which was fundamental in aiding the code to resolve expected ambiguities such as the marginalia that had chosen to ignore for this project, or the interchangeability of ‘u’/‘v’ as well as the distinction between the ‘f’/‘ſ’ characters common in early seventeenth-century print. This model was able to account for letter exchanges quite adeptly, making some general assumptions for the most likely candidate depending on word position. While this assumption is necessarily generalizing and thus incorrect, it minimized CER by accounting for Spanish word morphology and likely character combinations. This approach would inevitably require some human correction at the conclusion of the transcription, but the consistency the general assumption provides eased the complexity of training the AI. Some cases were easy to code for, such as the use of ‘ç’ or cedilla, which was no longer used after an orthographic reform in the eighteenth century, replaced by ‘z’.

In cases such as Figure 7, the assumption of ‘z’ whenever a ‘ç’ was present posed no difficulty other than the correct OCR result with a simple substitution applied afterwards. The model did not output “çar, como dixo Ciceron”, but “zar, como dixo Ciceron,” the desired result of the renAIssance team. When it came to ‘u’/‘v’ interchangeability, however, the process was more complex, such as with the example in Figure 8.

Figure 7:

Extract from Padilla’s Nobleza Virtuosa, p. 198.

Instead of transcribing this short passage incorrectly as “vno inconftante ceffando, o necio fi” in Figure 8, the algorithm was able to account for initial position ‘u’/‘v’ assuming a higher likelihood of ‘u’ instead of ‘v’, as well as intra-word ‘ſ’ as ‘s’ in most likely scenarios. Thus, the code correctly transcribed “uno inconstante cessando, o necio si” in this instance. This adaptability is a direct result of the use of TrOCR applied to a CNN, given this method’s aforementioned flexibility to recognize text in different conditions. A similar versatility was observed when it came to ‘ſ’/‘s’ interchangeability.

Figure 8:

Extract from Padilla’s Nobleza Virtuosa, p. 198.

The model was trained to assume intra-word ‘f’/‘s’ as likely ‘f’, while at the initial of final position of a word the character should most likely be ‘s’ given how pluralization works in Spanish as well as the minimal number of words that end in ‘f’. As seen in Figure 9, the model correctly outputs “stro” for the first word (itself the second component of a word hyphenated in the previous text line) and “antes” for the last one. However, the expectation for the second and third words would have been an output of “gufto” and “confiderando”, yet they were correctly rendered as “gusto” and “considerando.” The multiple rounds employed to pre-train and subsequently train the ML algorithm made their usefulness evident in this case, as the code learned from the context and frequency of certain words and phrases. Despite the assumption about ‘f’/‘s’ positioning, the TrOCR enhanced CNN was able to spell the words correctly because it recognized that “gusto” and “considerando” are far more likely in the given context than their alternative forms. This suggests that the fine-tuning process allowed the model to balance initial assumptions with learned linguistic patterns, leading to accurate predictions even in cases where the assumptions might have led to errors. This approach achieved a CER of 0.024 and WER of 0.047, indicating an excellent level of accuracy at 97.5 % for characters and a similarly high 95.22 % accuracy at the word level.^[7] Table 1 compares the CER, WER, and accuracy values of this approach with the other three explored by the renAIssance team.

Figure 9:

Extract from Padilla’s Nobleza Virtuosa, p. 198.

As the variability of the dataset progressively increased, this model encountered an obstacle caused by the initial assumptions on out-of-frame marginalia with which it was coded: the algorithm was trained to ignore marginalia and instead focus on the main framed text (see blue/green lines in Figure 6), as this was the printing style used in all of Padilla’s texts. However, upon expanding the dataset to a broader set of source texts as detailed in the Methods section, the model continued to apply this assumption to texts that had no frame or marginalia. Seeing neither of these two parameters, the model opted to ignore the last word of each line of text, expecting it to be text outside the frame as if it were marginalia. This was not an error by the model: the obstacle stemmed from the assumptions with which the code was created: these were accurate for the Padilla texts, but not applicable as a general principle to transcribe a more heterogenous corpus of archival materials. The solution was straightforward: the model was trained to differentiate whether there is a frame encasing the main text as a trigger to account for marginalia, making the algorithm incrementally more generalizable without sacrificing specialized performance achieved on the existing dataset.