Glosbe

Overview

This corpus contains dictionary and translation-memory entries collected from the crowdsourced online dictionary Glosbe. The current public corpus includes Amis, Atayal, Truku, and Saisiyat data. Most entries are translated into English, with an additional Amis-Mandarin collection.

---

Corpus Statistics

	Amis unknown	Atayal unknown	Truku	Saisiyat
Word count	89,909	577	116	600
Total audio	0	0	0	0
Transcribed	0	0	0	0
Untranscribed	0	0	0	0
Translated words
English	59,287	577	116	600
Mandarin	30,622	0	0	0
Japanese	0	0	0	0
Dutch	0	0	0	0
Morphologically segmented	0	0	0	0
Glossed words	0	0	0	0

---

Corpus Processing

The eight published XML files separate lexical entries from translation-memory entries and are organized by ISO 639-3 language code. The original Glosbe crawl is a retained snapshot and cannot be re-run; everything downstream of the published original tier is regenerated by CodeAndDocs/make_xml.sh, which runs three steps:

1. Cleaning and quote correction (clean_xml.py): normalizes punctuation and Unicode debris in the original tier and translations without changing the source spelling, and applies the Amis apostrophe-vs-quotation-mark correction described under Corpus Notes.

2. Standardization (standardize.py, once per language): rebuilds the kindOf="standard" FORM tier in FormosanBank's Ortho113 orthography using per-language conversion tables; stress accents (e.g. on Truku dálix) are deleted from the standard tier only, while the original tier keeps them exactly.

3. Phonology (add_phonology.py, once per language): generates IPA PHON tiers for both the original FORM (from the source orthography) and the standard FORM (from Ortho113).

Where one source form has several distinct translations, the additional ones are kept as separate TRANSL elements marked ver="alt". This includes the restored Amis-Traditional Chinese translation memory, rebuilt reproducibly from a reviewed conversion contributed by Joseph Lin.

---

Corpus Notes

Because this is a crowdsourced dictionary and translation memory, the linguistic accuracy and the rights attached to individual entries can vary. Treat entries as leads for review rather than as expert-verified dictionary records.

• In the Amis files, the letter ' (the glottal stop) is also used by the source as a quotation mark. Disambiguating apostrophe vs. quotation mark in the Amis data was done automatically: a classifier compares each sentence against its translations and an attested-word list and rewrites apostrophes it judges to be quotation marks to ". Occasional false positives are possible and misses are certain — some real quotation marks remain written as ', and a few glottal-stop letters may have been rewritten to ".

• Every such correction is logged with the full before/after sentence context in CodeAndDocs/quote_corrections.csv; the XML as it stood before any automated correction is preserved in CodeAndDocs/pre_correction_snapshot/.

• Glosbe entries generally do not identify a dialect, and a single file likely mixes dialects; the standardization assumptions per language are documented in the corpus README.

---

Access Details

• The repo containing the Glosbe corpus in FormosanBank as well as the code to reconstruct the corpus can be found here.

---

Copyright

The current XML records Glosbe and its contributors or source corpora as rights holders. The material was collected for private research review, and redistribution remains subject to Glosbe's terms and any third-party source rights. This is not a blanket Creative Commons corpus.

---

Citation

In accordance with our Terms of Use, if you use this corpus or any product derived from this corpus in any publication, you must cite both FormosanBank and:

• Glosbe. (2026). Glosbe [language]-[translation language] dictionary and translation memory. Cite the language pair, URL, and retrieval date recorded in the XML file used.