OLAC Language Extension

Copyright © 2007 Gary Simons (SIL International) and Steven Bird (University of Melbourne and University of Pennsylvania). This material may be distributed and repurposed subject to the terms and conditions set forth in the Creative Commons Attribution-ShareAlike 2.5 License.

Table of contents

1. Introduction
2. The ISO639-3 extension
3. Looking up the language for a code
4. Looking up the code for a language

References

1. Introduction

Identifying the languages involved is an important dimension of language resource description. However, using the character-string representation of language names as identifiers is problematic for several reasons:

• Different languages (in different parts of the world) may have the same name.

• The same language may have different names in different languages.

• The same language may have different names in the various countries where it is spoken.

• Within the same country, the preferred name for a language may change over time.

• In the early history of discovering languages (before their names were standardized), different people referred to the same language by different names.

• For languages having non-Roman orthographies, the language name may have several possible romanizations.

These facts taken together mean that identifying languages by name will not work. Rather, what is needed is a standard based on unique identifiers that do not change, combined with accessible documentation that clarifies the particular speech variety denoted by each identifier. For a deeper discussion of these issues see [CS2000] and [Simons2000].

The information technology community has a well-established standard for language identification, namely, ISO 639. Part 1 of the standard specifies two-letter codes for identifying about 160 of the world's major languages; part 2 specifies three-letter codes for identifying about 380 languages [ISO639-2]. These code sets in turn form the core of the standard followed by the Internet Engineering Task Force (IETF), namely, RFC 3066 [RFC3066]. This is the standard used for language identification in the xml:lang attribute of XML [XML-lang]. The Dublin Core Metadata Initiative [DCMT] defines encoding schemes dcterms:ISO639-2 and dcterms:RFC3066 for use in its Language element.

The above standards fall short of the coverage required by the language archiving community since they focus on the major languages of the world that are most frequently represented in the total body of the world's literature. A new part 3 of the standard [ISO639-3], adopted in 2007, has the purpose of defining three-letter identifiers for all known human languages. With over 7,500 codes, it attempts to provide a comprehensive enumeration of languages, including living, extinct, ancient, and constructed languages, whether major or minor. This is the standard that forms the basis for the OLAC recommendation on language identification in language resource description.

2. The ISO639-3 extension

The complete vocabulary of language identifiers recommended for use by OLAC consists of all active codes from ISO 639-3 as documented at its official web site [ISO639-3]. This includes both individual language identifiers and macrolanguage identifiers [ISO639-Macro]. At any time the valid code set is the complete set of codes (both active and retired) that is downloadable from [ISO639-3]; the use of retired codes is deprecated, but not disallowed.

Following the extension mechanism defined in [OLAC-Metadata], a language identifier is expressed as the value of the olac:code attribute and the extension olac:ISO639-3 is named in the xsi:type attribute. A language identifier may be used with the <dc:language> element to identify a language that a resource is written or spoken in. Similarly, a language identifier may be used with the <dc:subject> element to identify a language that a resource is about. In both cases, free text in the element content may be used to identify the specific variety of the language. For instance, the following indicates that a resource is written in Japanese:

<dc:language xsi:type="olac:ISO639-3" olac:code="jpn"/>

While the following indicates that a resource is sbout the Lau language of Solomon Islands, and specifically, the Suafa dialect:

<dc:subject xsi:type="olac:ISO639-3" olac:code="llu">Suafa dialect</dc:subject>

The formal definition of the vocabulary is in the following XML schema which conforms to the conventions for an OLAC metadata extension as defined in [OLAC-Metadata]:

http://www.language-archives.org/OLAC/1.1/olac-ISO639-3.xsd

That schema in turn includes another schema that contains only the simple type that enumerates all the recognized code values, namely,

http://www.language-archives.org/OLAC/1.1/ISO639-3.xsd

The latter schema provides a complete list of all language identifiers recognized by OLAC. Each enumerated value provides both a code (in the value attribute) and an associated language name (in the <annotation> element) that can be used for display purposes.

3. Looking up the language for a code

Given a particular three-letter code, the ISO 639-3 web site provides a means of looking up documentation for what the code represents. The three letters are appended to the documentation page URL as the value of the id parameter, as in:

http://www.sil.org/iso639-3/documentation.asp?id=abc

In addition to the basic information available on this page, more detailed information is available through a link to the corresponding page on the Ethnologue web site [Ethnologue] for a living or recently extinct language, or on the Linguist List website for an ancient [LL-Ancient] or constructed [LL-Constructed] language.

4. Looking up the code for a language

The easiest way to find the code for a living or recently extinct language is to type the name of a language or dialect or region into the Ethnologue's site search form at:

http://www.ethnologue.com/site_search.asp

When you know the country the language is spoken in, another approach is to use the country index to find a listing of all the languages in that particular country and then to browse the list:

http://www.ethnologue.com/country_index.asp

Linguist List offers a language search page based on name and country information downloaded from [Ethnologue-Codes] plus similar information they have compiled regarding ancient and constructed languages:

http://www.linguistlist.org/forms/langs/find-a-language-or-family.html

See [ISO639-3-Changes] for a description of the process by which changes are made to the international standard. Follow this process if you believe that something is missing or in error. Note that the standard reserves codes qaa through qtz for local use. That is, those codes will never be assigned as language identifiers. Thus, when users feel that a needed code is missing from the code set, they may use a local use code in their own database as a temporary measure until the outcome of a change request is known. Note, however, that local use codes are undefined for information interchange and should not be submitted to the OLAC harvester.

References