Wikipedia Monolingual Corpora
Here you can download text corpora extracted from the Wikipedia dumps in 23 languages, amounting to more than 5 billion tokens alltogether. Each XML file contains the full textual content of the individual language version of Wikipedia, extended with many annotations like article and paragraph boundaries, number of links referring to each article, crosslanguage links, categories and more. Have a quick look at a sample XML file containing one English article.
|Wikipedia XML file||zipped file size||language||number of articles||number of paragraphs||number of tokens|
XML Format Description
The XML files contain the following information:
|XML element||mother element||description|
|article||wikipedia (XML root)||This element encloses each Wikipedia article. It has an attribute name which contains the article’s title. The title is unique (in the current language version).|
|redirects||article||Articles that are redirects to another article are not stored in the XML files. However, the redirect information is contained in the redirects element of the target article. The attribute name contains a list of all article titles that redirect to the present article, separated by the pipe symbol (|).|
|links_in||article||The attribute name contains the total number of textlinks in other articles that link to the current article.|
|textlink||article||The attribute name contains the anchor text that is used in a textlink to refer to the current article. The attribute freq contains the number of times this anchor text was used to refer to the current article.|
|categories||article||The attribute name contains the list of Wikipedia categories the current article is assigned to. The individual categories are separated by the pipe symbol (|).|
|links_out||article||The attribute name contains the number of textlinks in the current article that refer to other Wikipedia articles.|
|crosslanguage_link||article||The attribute name contains the title of a Wikipedia article in another language the current article is linked to. The attribute language specifies the target language.
Important note: In Wikipedia, crosslanguage links may link an article to a section of a target article. In this case, the link contains the article name followed by a ‚#‘ and the section’s anchor name. These „deep“ links have been replaced by links to the whole article, i.e. the suffix starting at ‚#‘ has been deleted.
|disambiguation||article||This elements marks the article as a disambiguation page. It has no attributes.|
|content||article||This element encloses the current article’s textual content.|
|p||content, h, math, table||This element marks a paragraph boundary.|
|link||content, h, math, table||Marks a textlink to another Wikipedia article. The title of the target article is contained in the attribute target.|
|h||content||Marks a heading.|
|math||content||Marks a math formula.|
|table||content||Marks a table.|
|cell||table||Marks a table cell.|
Sample XML file
To see an example of the XML format click here.
The sample contains one article from the English Wikipedia.
Extracting raw text from XML
You can extract text only from the XML files using the Perl script xml2txt.pl. The usage is:
perl xml2txt.pl [Options] INPUT OUTPUT
where INPUT is an unzipped Wikipedia corpus XML file, and OUTPUT is the raw text file that will be produced. The encoding of the output file will be UTF-8.
|Options:||-articles||The article mark-up is preserved (<article name=“…“>…</article>).|
|-p||The paragraph mark-up is preserved (<p>…</p>).|
|-h||The headings mark-up is preserved (<h>…</h>).|
|-nomath||All content that is enclosed in math tags is deleted.|
|-notables||All content that is enclosed in table tags is deleted.|
|-nodisambig||Articles that are marked as disambiguation page are deleted.|
|-exclude-categories FILE||All articles that belong to one of the categories listed in FILE are ignored. FILE has one category name per line. Some useful categories are given below.|
|-only-categories FILE||Only articles are output that belong to one of the categories listed in FILE. FILE has one category name per line. Some useful categories are given below.|
Known and unknown bugs
The corpora are based on the Wikipedia dumps which contain articles in Wiki markup format, packed in XML. Wiki markup uses all kinds of brackets to mark links, categories, etc. Because articles can be edited manually by anybody, unmatched brackets sometimes occur. In order to minimize noise in the corpus, we discard all articles with unmatched brackets, i.e. articles that can’t be parsed.
We also (try to) discard sections with weblinks and references at the end of articles, because they often contain foreign language material. (It’s not a bug, it’s a feature!)
Main purpose of the Wikipedia Monolingual Corpora is to provide large text corpora in many languages which can be used for the routine tasks of corpus linguistics, like generating word frequency lists or collecting n-gram statistics.
However, the rich annotations in the XML files facilitate many more applications:
- categories allow to compile domain-specific corpora
- compile multilingual document-aligned comparable corpora using the crosslanguage links
- textlinks and redirects allow to collect expressions that are used to refer to a concept (i.e. a Wikipedia article)
- The annotations cover all requirements to build ESA style semantic similarity resources.
If you’d like to stay informed about corpora updates and new tools for text analysis you can subscribe to linguatools newsletter by providing your email address.