The World Digital Library

1 Background

Librarian of Congress James H. Billington first put forward the idea of a World Digital Library in a June 2005 speech at Georgetown University to the U. S. National Commission for UNESCO. The United States was at that time rejoining UNESCO after a nearly two-decade absence, and the commission was looking for projects that U. S. institutions could bring to UNESCO that would contribute to the ongoing revitalization of the organization. Billington proposed a cooperative project, to be undertaken by the Library of Congress and UNESCO in cooperation with partner libraries from around the world, to digitize and make freely available online primary source documents that tell the stories and highlight the achievements of all countries. Such a project, he argued, “would hold out the promise of bringing people closer together precisely by celebrating the depth and uniqueness of different cultures in a single global undertaking.”

UNESCO embraced the idea, which it saw as contributing to the achievement of a number of its own objectives, which included promoting knowledge societies, building capacity to exploit information and communications technology in developing countries, narrowing the digital divide between and within countries, and encouraging multilingualism and increased diversity of cultural content on the Internet. As a first step toward implementing Billington’s proposal, UNESCO and the Library of Congress jointly convened a meeting at UNESCO headquarters in Paris in December 2006 to solicit input from librarians and technology experts from many countries.

The main outcome of this meeting was a decision to establish working groups to plan different aspects of the project, especially technical architecture and content selection. A best practice working group co-sponsored by the Library of Congress and the International Federation of Library Associations and Institutions (IFLA) also was established. It was further agreed that the Library of Congress would develop, in consultation with the working groups and in cooperation with a select number of partners, the prototype of a future World Digital Library for presentation to the UNESCO General Conference in October 2007. Six partner institutions – the National Library of Brazil, the Bibliotheca Alexandrina of Alexandria, Egypt, the National Library and Archives of Egypt, the National Library of Russia, the Russian State Library, and the Library of Congress – joined in this effort and were present in Paris to unveil the prototype.

The next stage was to develop and launch the public version of the web site. Following eighteen months of intensive planning and development, www.wdl.org was officially unveiled on April 21, 2009 at UNESCO headquarters in Paris. Institutions from eighteen countries contributed content to the first public version of the WDL. Just over a thousand library items – books, manuscripts, maps and atlases, prints and photographs, sound recordings, and films – were included. International interest in the project, generated by coverage in both traditional and online media, was intense, as witnessed by the 587,510 visitors to the site on the launch day and the 619,846 visitors on the following day, April 22.

The project has continued to grow. As of this writing (March 2016), the WDL contains 13,551 library items, comprising 658,155 image and audio files. This content has been provided by 131 partner institutions in 58 countries. The majority of the partner institutions are libraries (103), followed by museums (9), archives (4), and other institutions (15) such as historical societies. In addition to the content-contributing partners, a number of other institutions and organizations – IFLA, Francophonie, the King Abdullah University of Science and Technology (KAUST) of Saudi Arabia, and UNESCO itself – are WDL partners, and contribute in other ways, e. g., by convening meetings and by helping to set standards and recruit new content-providing partners.

Governance of the WDL is carried out under a charter adopted by the partner institutions in 2010. The charter provides for a seven-member Executive Council that is charged with giving “leadership and direction to the WDL.” Five members of the Executive Council are elected from among the members; the other two members are the Director General of UNESCO (or his or her designate) and the head of the institution serving as WDL Project Manager. The five elected members of the Executive Council for the 2016–2020 term are the heads (or their designees) of the Bibliotheca Alexandrina, the Bibliothèque nationale de France, the National Library of Brazil, the National Library of China, and the Qatar National Library. Dr. Ismail Serageldin, director of the Bibliotheca Alexandrina, was elected chairman of the Executive Council in 2010, and reelected to this post in 2015. Under the terms of the charter, the Library of Congress was designated the Project Manager for the period 2010–2015. By decision of the Executive Council at its meeting in Alexandria, Egypt in November 2015, the Library of Congress’s term as Project Manager has been extended for another five years, up to 2020.

New partner institutions join the WDL by applying to the Project Manager and, upon its recommendation, being approved by the Executive Council. They then accede to the WDL Charter. The WDL ultimately seeks to have at least one content providing partner in every country in the world. Joining the project is cost-free – there are no charges or membership fees – but partner institutions must bear the cost of digitizing their own content for contribution to the WDL. In many parts of the world – Europe, North America, East Asia, much of South America, and the more affluent parts of the Middle East – this is not a problem, as cultural institutions already have digitization programs for their own institutional, national, and regional digital library projects. Elsewhere, however, many libraries and other heritage institutions lack the equipment and trained personnel needed to digitize their collections. The WDL has been able to provide some financial and technical assistance to libraries in developing countries that lack the means to establish digitization efforts (most notably by funding the establishment of digital conversion centers at the national libraries of Egypt, Iraq, and Uganda), but capacity building for digital libraries remains a major challenge in many countries. The WDL is keen to work with bilateral and multilateral partners and to contribute to cooperative international efforts involving, for example, UNESCO, IFLA, Francophonie, and other organizations with the objective of building digital library capacity in developing countries.

Fig. 1:

Moving digitization equipment provided by the WDL into the National Library of Uganda, 2012

The Library of Congress has spent approximately $20 million in developing and maintaining the WDL over the last decade. This funding has come from external sources rather than U. S. government appropriations, with the largest contributors so far being Carnegie Corporation of New York, the Qatar National Library of Qatar Foundation for Education, Science and Community Development, Google, the National Digital Library Trust Fund of the Library of Congress, the James Madison Council of the Library of Congress, Microsoft, and KAUST. Fundraising for the project remains a major ongoing challenge, given the existence of many needs and competing priorities at the Library of Congress and elsewhere.

2 Content

The main distinguishing feature of the WDL is quality: both the quality and importance of the cultural and historical artifacts that it displays and the high level of technical and curatorial care with which these documents are presented. Among the more striking examples of artifacts on the WDL are the following:

1. The Florentine Codex (General History of the Things of New Spain), Mexico, 1577, Medicea Laurenziana Library, Florence, www.wdl.org/10096

2. Zakarīyā ibn Muhammad al-Qazwīnī, The Wonders of Creation, Iran, circa 1500–1599, National Library and Archives of the Islamic Republic of Iran, www.wdl.org/10606

3. Ludwig van Beethoven, Symphony Number 9 in D Minor, Opus 125 (autograph score), Germany, 1828, Berlin State Library, www.wdl.org/15063

4. Codex Aureus, United Kingdom, circa 750, National Library of Sweden, www.wdl.org/17185

5. 坤輿萬國全圖 (Great Universal Geographic Map of Matteo Ricci), China, 1602, University of Minnesota Library, www.wdl.org/4136

6. Frances Crick, “Sketch of the DNA Double Helix,” United Kingdom, 1953, Welcome Library, www.wdl.org/3252

7. Early recording of the Marseillaise, France, 1898, Bibliothèque nationale de France, www.wdl.org/3017

8. Ancient Greek text, The Curse of Artemisia (fragment), Egypt, circa 350–351 BCE, Austrian National Library, www.wdl.org/4310

Fig. 2:

Pages from the Florentine Codex and the Wonders of Creation

Under an agreement with UNESCO, the WDL offers to host digital copies of all documents and collections inscribed on the UNESCO Memory of the World register on www.wdl.org, and it actively solicits such collections for the site. At present, 675 documents that are registered on Memory of the World as individual items or parts of collections are displayed on the WDL. These documents have been contributed by institutions in Austria, Belgium, Brazil, Germany, Italy, Mexico, Norway, Peru, Portugal, Serbia, Slovakia, South Korea, and the United States.

Not everything in the WDL, to be sure, is a centuries old, one-of-a-kind or extremely rare document. There are also many books, published maps, prints and photographs, as well as films and sound recordings dating from the nineteenth and early twentieth centuries, many of which exist in multiple copies around the world. Such works are carefully chosen, however, and are often parts of larger bodies of content, e. g., photographs and photograph albums that collectively make up the earliest photographic surveys of the Chinese and Russian empires (contributed by the Library of Congress and Yale University); the many books in Arabic published in Cairo, Beirut, and Istanbul in the nineteenth century that reflect publishing and printing in the period of what has been called the “Arab Renaissance” (contributed by the Qatar National Library and several other partners); or the large and growing collection of books, maps, and periodicals from Afghanistan and surrounding countries that are being added to the WDL in an effort to “digitally repatriate” to Afghanistan materials relating to its history that have been destroyed by war (funded by Carnegie Corporation of New York).

Other subject areas in which the WDL has been building world-class concentrations of content are Mesoamerican codices that illustrate the Aztec and Mayan heritage of Mexico; Arabic scientific manuscripts; Chinese rare books and manuscripts; and Bibles, Qur’ans, and other sacred texts. A total of 131 languages are represented on the WDL, ranging from thousands of items for such world languages as Arabic, English, and Spanish, to one or a few items for such rare and endangered languages as Achagua, Chibcha, and Sáliba (indigenous languages of South America).

3 The WDL User Interface: Features and Functionality

The features and functionality of the WDL website were largely determined during the 2006–2009 planning phase carried out within the UNESCO-Library of Congress working groups and at the Library of Congress. Since the launch of the public website in 2009, there have been steady incremental improvements, some quite significant (e. g., the introduction of full text search for books in seven languages), and one fundamental redesign of the User Interface that was completed in November 2014. The latter involved a shift to “responsive design” that is intended to accommodate the growing variety of devices used by those accessing the WDL including, most importantly, mobile telephones. For the most part, however, the basic principles that inform the WDL’s features and functionality were set during the planning process. They are (1) multilingualism; (2) use of consistent, high quality metadata; (3) provision of expert curatorial interpretation of the content; and (4) high performance with regard to speed, accessibility, and options for use and re-use.

Multilingualism. In order to create a true world digital library and to justify the expense and effort that it would entail, it was essential for the WDL to focus on multilingualism. The WDL could not be a site offered in just one or two European languages, e. g., English and French, or a site that was primarily in English but that offered certain features in other languages, or even one that offered convenient links to commercial machine translation tools. Rather, it had to be truly multilingual, in the sense that it would offer an equivalent user experience to visitors accessing the site in different languages. This implied an ambitious translation model, one in which all metadata and navigational information was translated into each interface language. This meant reliance on human machine-assisted translation, rather than machine translation in order to ensure correct and sophisticated handling of complex cultural terms.

It is important to note that the language that appears on a WDL user’s computer/tablet/mobile is the “native” language of that user’s browser. Someone accessing www.wdl.org in Brazil automatically sees a site that is Portuguese. Most people who access the WDL – especially those that are sent via search engines to particular items, without going first to the home page – are unaware that the WDL is available in six other languages and that they can switch between languages at will. To users in Brazil, the WDL is a Portuguese website and digital library. What this means, of course, is that users expect a very high level of linguistic competence and quality, and it is the need to strive for and attain this level of quality that drives the WDL translation process.

Fig. 3:

Item level display in Portuguese

After a period of considerable internal debate, it was decided to offer interfaces in seven languages – the six official UN languages (Arabic, Chinese, English, French, Russian, and Spanish) plus Portuguese. Providing interfaces in these languages enables a large portion of Internet users worldwide to access the WDL in their native languages. The selection was by no means perfect, however, and choice of these seven does leave some important gaps in the WDL’s global coverage. For this reason, the addition of other languages in the future is not ruled out. The biggest gap in the WDL’s linguistic coverage is in South and Southeast Asia, where hundreds of millions of speakers of Hindi and Urdu cannot be reached in their native languages. In Europe, the omission of German misses a large audience; usage in Germany, where many people already access the WDL in English (and where several major libraries are important WDL contributors) would increase dramatically were there a German interface. Much the same is true in Japan, Italy, Iran, and a few other countries. Addition of new languages (which would require the retroactive translation of several million words of text for each new language, as well as a commitment to support the continued provision of translation in each language going forward as the WDL expands) would be an enormously expensive effort, and most likely would have to be done using innovative methods such as crowdsourcing (with which the WDL is already experimenting on a limited basis).

Consistent, High Quality Metadata. The importance of consistent, high quality metadata follows to some extent from the translation model that was adopted. Providing an equivalent user experience across languages requires that subject headings and proper names be consistently translated across interfaces. For a French-speaking user to find the same set of results for, e. g., “horse,” as his English-speaking counterpart requires ensuring that “horse” is consistently translated into “cheval” and not something else. Variant terms, such as “steed,” “stallion,” “mare,” “mount,” “charger,” and so forth add complexity, as do plurals, the complexities of highly inflected languages such as Russian, diacritics, and the four variant alphabets/character systems required: Arabic, Chinese, Cyrillic, and Latin. Full text search in seven languages, such as the WDL provides for books, creates additional difficulties. To overcome these challenges has required a steep learning curve since 2009 and the development and use of software tools that include translation memories, glossaries, and a cataloging application developed specifically for the WDL.

Fig. 4:

Search Results Page in French

Beyond the demands set by the translation model, consistent, high-quality metadata is needed to ensure that the WDL is technically able to meet its express goals of encouraging visitors to the site not only to explore content from their own country and in their own language, but to engage in cross-national and cross-cultural comparisons, and to do so for different historical periods. Every item in the WDL is cataloged for six main values that drive the site’s core browse options, and that can be used singly or in combination with each other (and in combination with search):

1. place (i. e., place as subject, or what the item is about, not place of creation or publication, which may or may not be the same. Place of creation/publication is of course also coded, but it does not drive a browse option)

2. date (i. e., of creation. Date as subject matter is also coded, but it does not drive a browse option)

3. topic (based on a modified Dewey Decimal system, supplemented by “additional subjects” which are displayed as facets)

4. format

5. language (i. e., of the document, if it has linguistic content)

6. contributing institution.

Fig. 5:

WDL Browse Options

Browse and search results can be viewed in three ways: list (which provides a preview of the item-level description); gallery (which shows just the reference images for the content, providing a very striking visual effect and a quick way to page through a large number of items); and map (which plots the items found according to the “place” value in the bibliographic record). The map view option, combined with the metadata that underpins it (geocoding by latitude and longitude) provides a powerful tool for showing items that cluster according to geographic patterns (e. g., albums showing the construction of railroads across China, North America, or Siberia, which, provided the geocoding is sufficiently precise and accurate, actually show the route of the railroad and construction sites along the way).

These facets provide a set of tools for browsing and searching across cultures. Under the “Browse by Time” option, for example, the period “1200–1299” yields a total of seventy items (as of March 2016). The “Narrow results” button can be used to give a preview of what these items are in terms of format (58 manuscripts, eighteen books, and one print), which institutions provided them, the places with which they are associated, and the language of the content. With regard to the latter, the metadata reveals that of these seventy thirteenth-century documents, eighteen are in Chinese (all of the books), thirteen are in Arabic, twelve are in Latin, and six are in Church Slavic. Also seen are items in Middle High German, Old French, Catalan, and Old Provençal, reflecting the rise in Europe of the vernacular languages, which at about this time were taking their place alongside Latin.

This linguistic distribution is about what one would expect from what is known about which parts of the world at this time were producing large bodies of written documentation: China, the Islamic world, and both Western and Eastern Europe, something that becomes visually quite apparent when these seventy items are displayed on the map view. But the plotting on the map turns up interesting geographic outliers that might provoke thought and inspire discussion. The Western hemisphere shows a single pin, located in southern Mexico, where the language Yucateco is plotted. This indicates the flourishing Mayan culture of this period, which existed contemporaneously with, although of course entirely unknown to, the civilizations of late Middle Age Europe, Song dynasty China, and the Mamluk Caliphate of the Middle East. Similarly, a point on the southwestern tip of the Arabian Peninsula plots a manuscript in the language of Judeo-Yemeni Arabic. This attests to the presence of the Jewish communities of Yemen, living alongside and culturally very much intertwined with the Muslim communities of the same region, that even now are living out the final chapters of their long historical existence.

Fig. 6:

Browse results from 1200–1299 plotted on the map display

One challenge presented by all this high-quality metadata, to be sure, is that of encouraging visitors to the WDL site to actually use it. Evidence suggests that online users – or even visitors to physical libraries – particularly those born in or who have come of age in the digital age, have been conditioned by Google and other search engines and by sites such as Amazon to go straight to the search box. They have little familiarity with or patience for the facets based on the traditional library metadata on which the WDL and indeed most national and international digital libraries are built. One challenge of user outreach, and an area for cooperation with teachers and schools, is to better familiarize users with facets and other browse and search tools, and to teach them that they can more easily find what they are looking for if they take advantage of metadata in which librarians have made such enormous investments going back decades if not centuries.

Curatorial Interpretation. In addition to being provided with consistent, high quality metadata, every item on the WDL contains a paragraph-length description that explains what the item is and why it is important. These descriptions are written by scholars and curators, and provide context that goes beyond the basic metadata. The descriptions play an essential role in helping the WDL to fulfill its mission of encouraging understanding and appreciation across cultures. In order for a European visitor to have some appreciation of an Arab or Persian manuscript, he or she has to be told something about the content and provenance of that manuscript – to be provided with answers to two key questions: what is this item? Why does it matter and why should I care about it? The item-level descriptions also should perform the function of encouraging users to actually look at the content that is being presented to them, even though at first glance it may seem alien and impenetrable. This can be done by pointing out particular illuminations or illustrations, or simply by providing information about the item that kindle interest and curiosity: that it is the first or oldest of some category; that it belonged to a king or queen; that it was carried off by the Vikings; or that it was confiscated as war booty or enemy property at the end of this or that conflict.

In addition to the item-level descriptions, the WDL offers a number of other features that provide context and explanation. These include videos for selected items in which expert curators provide in-depth explanations about the items, timelines, interactive maps, and thematic sections dealing with broad, cross-national and interdisciplinary topics. One thematic section, devoted to the history of Arabic and Islamic science, already has been developed.

Other themes discussed by the WDL partners and the Executive Council in recent years have included the Mesoamerican codices of pre- and early post-Columbian Latin America; the expanding of frontiers to build continental-scale nation-states; sacred texts; the history of China; the building of the world’s great railroads, canals, and the infrastructure of modern transportation; systems of writing; and “intertwined” histories, e. g., travel and voyages, the Silk Road, epidemics, World War I as a global war, and the Atlantic slave trade. Further development of thematic sections and other interpretive features will be a major focus of the WDL in the coming years.

Performance. A final area in which the WDL seeks to be distinctive is performance. The most important performance measures are speed and reliability, which the WDL seeks to maintain by using cloud storage and a content delivery network (CDN). The more than one million visitors on April 21–22, 2009 – reaching tens of thousands of users per hour in peak periods – were handled flawlessly without a crash, largely because of the CDN.

Beyond speed and reliability, the WDL seeks to enhance the user experience through a number of other features. These include a text-to-voice conversion feature that can be used to read the metadata and descriptive information in all seven of the WDL languages and a state-of-the-art zoom feature that can be applied not only to prints, photographs, maps, and other pictorial elements, but to every page of every book, newspaper, and journal. Other useful features include downloadable content and near-exclusive reliance on color scanning rather than bitonal or greyscale images.

The commitment to these demanding quality and performance standards has necessarily limited the quantitative growth of the WDL, as measured by numbers of collections and items, which, particularly in the early years of the project, was frustratingly slow. It is important to emphasize, however, that in information science terms the WDL is not a “small” website. It is in fact very large, as the features it provides require vastly more data than is needed for national and international projects that focus on aggregating metadata from distributed websites. Behind the 13,500 items/650,000 files of the WDL are more than 220 million image tiles comprising nearly 2 terabytes of storage that are required to facilitate the zoom views on the site. The masters for these files comprise more than a million files, comprising more than 18 terabytes of storage. The WDL site uses 4,644,500 HTML 5 pages (94,500 item detail pages and 4,550,000 viewer pages), each of which has a unique URL. More than 650,000 files are available for download, and there are approximately 210,000 pages of searchable text (which rely upon 750,000 word coordinate and plain text files comprising 2.2 gigabytes of storage). Google crawls approximately 400,000 WDL pages. The metadata (including item level descriptions) comprises about 35 million words of text in the seven interface languages (Arabic, Chinese, English, French, Portuguese, Russian, and Spanish). The numbers are steadily growing, and are expected to increase in the coming years as the WDL expands.

4 Usage

The Library of Congress keeps detailed visitor statistics for the WDL, using two major tracking tools, Omniture and Google Analytics. The number of visitors to www.wdl.org since April 2009 totals nearly 41 million. In 2015, the WDL website recorded 8,185,301 visits. The top countries in the world by number of visitors were China, the United States, Brazil, Mexico, Spain, France, the United Kingdom, Germany, Russian Federation, and Canada. Usage by interface language was 33.2 percent Spanish, 23.1 percent English, 15.4 percent Chinese, 9.8 percent Arabic, 7.2 percent Portuguese, 5.6 percent French, and 5.6 percent Russian.

5 Priorities and Prospects for the Future

Moving forward, the WDL faces a number of challenges, and will be focusing on several priorities.

First, in line with its quest for universal participation, the WDL will continue to seek to recruit new partners with content to contribute, with emphasis on parts of the world that are currently un- or underrepresented in the WDL. The latter include Sub-Saharan Africa, Central America and the Caribbean, and South and Southeast Asia. In some of these regions, capacity building efforts are needed so that libraries and other heritage institutions can participate in a meaningful way, i. e., digitize their collections and contribute them to the project as well as develop their own institutional and national digital libraries.

Second, the WDL will continue adding content, while maintaining its current high quality standards. The goal is not to reach any particular numerical target, but rather to ensure reasonable representation across countries, languages, religions, cultures, and time periods – to make sure that no one comes away from the WDL feeling totally left out or unrepresented. This is of course an ambitious, long-term task.

Third, as noted above, we need to work with partner institutions – libraries, but also schools, universities, museums, and others – to produce timelines, interactive maps, thematic sections, essays, lesson plans – perhaps even MOOCs – that will ensure that the interesting and important content on the WDL is used in educational settings at all levels.

Finally, we will continue to build our global audience, with the aim of reaching tens of millions of people, on mobiles, tablets, and PCs across all seven of the WDL interfaces.