Wikipedia Monolingual Corpora

line

Here you can download text corpora extracted from the Wikipedia dumps in 23 languages, amounting to more than 5 billion tokens alltogether. Each XML file contains the full textual content of the individual language version of Wikipedia, extended with many annotations like article and paragraph boundaries, number of links referring to each article, crosslanguage links, categories and more. Have a quick look at a sample XML file containing one English article.

Download

Wikipedia XML file zipped file size language number of articles number of paragraphs number of tokens
arwiki-20140714-corpus.xml.bz2 189 MB Arabic 273,709 1,654,018 61,601,807
bgwiki-20140728-corpus.xml.bz2 138 MB Bulgarian 183,983 1,115,838 43,324,881
cswiki-20140730-corpus.xml.bz2 296 MB Czech 341,446 2,382,825 86,076,579
dawiki-20140725-corpus.xml.bz2 144 MB Danish 188,415 1,177,350 43,997,748
dewiki-20140725-corpus.xml.bz2 1,88 GB German 1,697,608 14,971,566 649,943,374
elwiki-20140728-corpus.xml.bz2 113 MB Greek 96,742 863,312 37,468,841
enwiki-20140707-corpus.xml.bz2 4,25 GB English 4,579,471 43,300,386 1,714,676,058
eswiki-20140810-corpus.xml.bz2 1,04 GB Spanish 1,065,798 9,934,311 428,385,578
fawiki-20140802-corpus.xml.bz2 156 MB Farsi 1,438,575 2,584,340 55,847,903
fiwiki-20140809-corpus.xml.bz2 258 MB Finnish 349,907 2,117,539 62,254,999
frwiki-20140804-corpus.xml.bz2 1,43 GB French 1,516,664 14,854,031 546,824,176
huwiki-20140727-corpus.xml.bz2 314 MB Hungarian 261,378 2,595,194 89,823,011
itwiki-20140810-corpus.xml.bz2 1,02 GB Italian 1,127,405 9,583,932 381,082,826
jawiki-20140807-corpus.xml.bz2 1,28 GB Japanese 1,049,338 10,017,880 45,124,304
kowiki-20140801-corpus.xml.bz2 217 MB Korean 281,130 1,803,057 50,312,632
nlwiki-20140804-corpus.xml.bz2 631 MB Dutch 2,147,989 6,637,191 235,814,312
plwiki-20140802-corpus.xml.bz2 715 MB Polish 1,220,833 6,651,210 208,709,602
ptwiki-20140806-corpus.xml.bz2 508 MB Portuguese 978,010 5,041,117 183,120,152
rowiki-20140729-corpus.xml.bz2 157 MB Romanian 247,651 1,561,772 52,490,661
ruwiki-20140727-corpus.xml.bz2 1,09 GB Russian 1,444,962 10,796,451 334,731,419
svwiki-20140818-corpus.xml.bz2 407 MB Swedish 1,790,146 7,051,083 166,218,170
trwiki-20140806-corpus.xml.bz2 175 MB Turkish 233,218 1,691,110 48,378,679
zhwiki-20140804-corpus.xml.bz2 552 MB Chinese 967,153 5,629,801 20,577,336

XML Format Description

The XML files contain the following information:

XML element mother element description
article wikipedia (XML root) This element encloses each Wikipedia article. It has an attribute name which contains the article’s title. The title is unique (in the current language version).
redirects article Articles that are redirects to another article are not stored in the XML files. However, the redirect information is contained in the redirects element of the target article. The attribute name contains a list of all article titles that redirect to the present article, separated by the pipe symbol (|).
links_in article The attribute name contains the total number of textlinks in other articles that link to the current article.
textlink article The attribute name contains the anchor text that is used in a textlink to refer to the current article. The attribute freq contains the number of times this anchor text was used to refer to the current article.
categories article The attribute name contains the list of Wikipedia categories the current article is assigned to. The individual categories are separated by the pipe symbol (|).
links_out article The attribute name contains the number of textlinks in the current article that refer to other Wikipedia articles.
crosslanguage_link article The attribute name contains the title of a Wikipedia article in another language the current article is linked to. The attribute language specifies the target language.
Important note: In Wikipedia, crosslanguage links may link an article to a section of a target article. In this case, the link contains the article name followed by a ‚#‘ and the section’s anchor name. These „deep“ links have been replaced by links to the whole article, i.e. the suffix starting at ‚#‘ has been deleted.
disambiguation article This elements marks the article as a disambiguation page. It has no attributes.
content article This element encloses the current article’s textual content.
p content, h, math, table This element marks a paragraph boundary.
link content, h, math, table Marks a textlink to another Wikipedia article. The title of the target article is contained in the attribute target.
h content Marks a heading.
math content Marks a math formula.
table content Marks a table.
cell table Marks a table cell.

Sample XML file

To see an example of the XML format click here.
The sample contains one article from the English Wikipedia.

Extracting raw text from XML

You can extract text only from the XML files using the Perl script xml2txt.pl. The usage is:

perl xml2txt.pl [Options] INPUT OUTPUT

where INPUT is an unzipped Wikipedia corpus XML file, and OUTPUT is the raw text file that will be produced. The encoding of the output file will be UTF-8.

Options: -articles The article mark-up is preserved (<article name=“…“>…</article>).
-p The paragraph mark-up is preserved (<p>…</p>).
-h The headings mark-up is preserved (<h>…</h>).
-nomath All content that is enclosed in math tags is deleted.
-notables All content that is enclosed in table tags is deleted.
-nodisambig Articles that are marked as disambiguation page are deleted.
-exclude-categories FILE All articles that belong to one of the categories listed in FILE are ignored. FILE has one category name per line. Some useful categories are given below.
-only-categories FILE Only articles are output that belong to one of the categories listed in FILE. FILE has one category name per line. Some useful categories are given below.

Useful categories

language description categories file
de people people_de.txt
en medicine medicine_en.txt

Known and unknown bugs

The corpora are based on the Wikipedia dumps which contain articles in Wiki markup format, packed in XML. Wiki markup uses all kinds of brackets to mark links, categories, etc. Because articles can be edited manually by anybody, unmatched brackets sometimes occur. In order to minimize noise in the corpus, we discard all articles with unmatched brackets, i.e. articles that can’t be parsed.

We also (try to) discard sections with weblinks and references at the end of articles, because they often contain foreign language material. (It’s not a bug, it’s a feature!)

Applications

Main purpose of the Wikipedia Monolingual Corpora is to provide large text corpora in many languages which can be used for the routine tasks of corpus linguistics, like generating word frequency lists or collecting n-gram statistics.

However, the rich annotations in the XML files facilitate many more applications:

  • categories allow to compile domain-specific corpora
  • compile multilingual document-aligned comparable corpora using the crosslanguage links
  • textlinks and redirects allow to collect expressions that are used to refer to a concept (i.e. a Wikipedia article)
  • The annotations cover all requirements to build ESA style semantic similarity resources.

License

The XML files that you can download above are derived from the original Wikipedia and are therefore made available under the same license as Wikipedia itself: Creative Commons Attribution-ShareAlike.

cc-by-sa


If you’d like to stay informed about corpora updates and new tools for text analysis you can subscribe to linguatools newsletter by providing your email address.