OLAC Language Vocabulary

Copyright © 2003 Gary Simons (SIL International) and Steven Bird (University of Melbourne and University of Pennsylvania). This material may be distributed and repurposed subject to the terms and conditions set forth in the Creative Commons Attribution-ShareAlike 2.5 License.

Table of contents

1. Introduction
2. The OLAC-Language extension
3. Looking up the code for a language

References

1. Introduction

Language identification is an important dimension of language resource description. However, using the character-string representation of language names as identifiers is problematic for several reasons:

• Different languages (in different parts of the world) may have the same name.

• The same language may have a different name in each country where it is spoken.

• Within the same country, the preferred name for a language may change over time.

• In the early history of discovering new languages (before names were standardized), different people referred to the same language by different names.

• For languages having non-Roman orthographies, the language name may have several possible romanizations.

The sum of these facts taken together suggests that a standard based on names will not work. Rather, what is needed is a standard based on unique identifiers that do not change, combined with accessible documentation that clarifies the particular speech variety denoted by each identifier.

The information technology community has a standard for language identification, namely, ISO 639 [ISO639]. Part 1 of this standard lists two-letter codes for identifying about 160 of the world's major languages; part 2 of the standard lists three-letter codes for identifying about 380 languages. ISO 639 in turn forms the core of the standard followed by the Internet Engineering Task Force (IETF), namely, RFC 3066 [RFC3066] (formerly RFC 1766 [RFC1766]). This is the standard used for language identification in the xml:lang attribute of XML [XML-lang] and in the language element of the Dublin Core Metadata Initiative [DCMT]. RFC 3166 provides a mechanism for users to register new language identification codes for languages not covered by ISO 639, but very few additional languages have been registered.

Unfortunately, the existing standard falls far short of meeting the needs of the language resources community since it fails to account for more than 90% of the world's languages, and it fails to adequately document what languages the codes refer to ([CS2000], [Simons2000]). In order to achieve complete coverage, OLAC makes use of SIL International's Ethnologue [Ethnologue] which provides a unique three-letter code for all known living languages, plus recently extinct languages. In order to extend coverage to all known languages, OLAC also uses the four-letter codes that have been developed by Linguist List to identify ancient [LL-Ancient] and constructed languages [LL-Constructed] that are not within scope for the Ethnologue. The extension mechanism of RFC 3066 is used in order to develop a means of referring to these codes that is compatible with the IETF's standard.

2. The OLAC-Language extension

The complete vocabulary of OLAC language identifiers consists of three sets of codes:

• all two-letter codes from ISO 639-1

• all three-letter codes from SIL's Ethnologue (prefixed by x-sil- to make them compatible with RFC 3066)

• all four-letter codes from Linguist List (prefixed by x-ll- to make them compatible with RFC 3066)

This complete vocabulary is defined in the following XML schema which conforms to the conventions for an OLAC metadata extension as defined in [OLAC-Metadata]:

http://www.language-archives.org/OLAC/1.0/olac-language.xsd

This schema in turn includes another schema that contains only the simple type that enumerates all the recognized code values. The code list itself is contained in:

http://www.language-archives.org/OLAC/1.0/LanguageCodes.xsd

The latter schema provides a complete list of all language identifiers recognized by OLAC. Each enumerated value provides both a code (in the value attribute) and an associated language name (in the label attribute) that can be used for display purposes.

3. Looking up the code for a language

The SIL Ethnologue [Ethnologue] provides approximately 7,000 three-letter codes, along with detailed information about language names, genetic affiliations, and geographical locus, amongst other things. The easiest way to find the code for a language is to type a language name into the site search form at:

http://www.ethnologue.com/site_search.asp

When you know the country the language is spoken in, another approach is to use the country index to find a listing of all the languages in that particular country and then to browse the list:

http://www.ethnologue.com/country_index.asp

Linguist List offers a form for searching for three-letter codes, given all or part of a language name:

http://www.linguistlist.org/forms/langs/find-a-language-or-family.html

A three-letter Ethnologue code AAA will be represented in the OLAC language vocabulary as x-sil-AAA.

To build a language search facility into your own software, you may download tables of codes, language names, alternate names, and countries where spoken from the Ethnologue web site [Ethnologue-Codes], and load them into a relational database using the schema described in [Simons2000].

If you do not find the language you are looking for in the Ethnologue and it is an extinct or constructed language, then consult the Linguist List code tables at [LL-Ancient] and [LL-Constructed]. The Linguist List search form cited above finds codes from both the Ethnologue and Linguist List sets. A four-letter Linguist List code XAAA will be represented in the OLAC language vocabulary as x-ll-XAAA.

See [Ethnologue-Codes] for instructions on how to communicate with the Ethnologue staff about changes that might be needed to the code set. Note that the codes QVA through QZZ are reserved for local use. That is, they will never be assigned by SIL International as language identifiers. Thus, when users feel that a needed code is missing from the code set, they may freely use a code from the local use range as a temporary measure until the outcome of a change request is known.

To do

References