How may Wiktionary's API be used to determine whether or not a word exists?
-
1Anyone who has read the documentation will see that the API contains nowhere near enough functionality to "retrieve Wiktionary word content". I'd estimate it gets you roughly 1% of the way. You can retrieve raw wiki syntax or parsed HTML and from there you have to do everything yourself. Having said that there might be a very new experimental API that works only on the English Wiktionary.– hippietrailAug 28, 2016 at 6:28
-
6Get all Wiktionary articles in individual JSON files here: github.com/dan1wang/jsonbook-builder– danielApr 13, 2019 at 9:05
-
2An even better parsed JSON version is here: kaikki.org– PuxMay 31, 2022 at 9:41
9 Answers
The Wiktionary API can be used to query whether or not a word exists.
Examples for existing and non-existing pages:
http://en.wiktionary.org/w/api.php?action=query&titles=test
http://en.wiktionary.org/w/api.php?action=query&titles=testx
The first link provides examples on other types of formats that might be easier to parse.
To retrieve the word's data in a small XHTML format (should more than existence be required), request the printable version of the page:
http://en.wiktionary.org/w/index.php?title=test&printable=yes
http://en.wiktionary.org/w/index.php?title=testx&printable=yes
These can then be parsed with any standard XML parser.
-
4Thanks; the API itself is not what I was hoping for but the link you provided is what I was looking for. May 14, 2010 at 2:19
-
2Now it accepts additional format parameter for other than xml output like so : en.wiktionary.org/w/…– eenagyJun 28, 2015 at 18:07
-
4Might not work as you expect though en.wiktionary.org/wiki/Category:English_misspellings en.wiktionary.org/wiki/amatuer– endolithApr 30, 2016 at 4:15
-
1If you need to fetch the data using the browser, you can use
https://en.wiktionary.org/w/api.php?format=json&action=query&origin=*&export&exportnowrap&titles=test
to avoid CORS-related problems May 21, 2019 at 0:49 -
5
There are a few caveats in just checking that Wiktionary has a page with the name you are looking for:
Caveat #1: All Wiktionaries including the English Wiktionary actually have the goal of including every word in every language, so if you simply use above API call you will know that the word you are asking about is a word in at least one language, but not necessarily English: http://en.wiktionary.org/w/api.php?action=query&titles=dicare
Caveat #2: Perhaps a redirect exists from one word to another word. It might be from an alternative spelling, but it might be from an error of some kind. The API call above will not differentiate between a redirect and an article: http://en.wiktionary.org/w/api.php?action=query&titles=profilemetry
Caveat #3: Some Wiktionaries including the English Wiktionary include "common misspellings": http://en.wiktionary.org/w/api.php?action=query&titles=fourty
Caveat #4: Some Wiktionaries allow stub entries which have little or no information about the term. This used to be common on several Wiktionaries but not the English Wiktionary. But it seems to have now spread also to the English Wiktionary: https://en.wiktionary.org/wiki/%E6%99%B6%E7%90%83 (permalink for when the stub is filled so you can still see what a stub looks like: https://en.wiktionary.org/w/index.php?title=%E6%99%B6%E7%90%83&oldid=39757161)
If these are not included in what you want, you will have to load and parse the wikitext itself, which is not a trivial task.
-
3What I really wanted to do was take a full dump of the data on one of the non-English Wikitionary sites, and then turn the contents into something I could use locally. It seems silly now, but I was hoping that I could request the list of all words, and then pull down their defitions/translations one at a time as needed. Dec 5, 2010 at 17:51
-
1The fix to Caveat #2 is simple: add
&prop=info
to the query and check the response forredirect
attribute.– svickApr 30, 2012 at 11:17 -
@svick: Yes it's true #2 is easier to circumvent when using the API but these basic caveats also cover trying to parse the Wiktionary data dump files, even though this question doesn't ask about that approach. Apr 30, 2012 at 11:26
You can download a dump of Wiktionary data. There's more information in the FAQ. For your purposes, the definitions dump is probably a better choice than the XML dump.
-
3Those dump files are massive, and it's unclear which ones to download (all of them?). Probably not what most people are looking for it they just want to programmatically lookup a handful of words.– CerinJun 14, 2012 at 18:25
-
1I explain which file to download - i.e. the definitions dump (the directory from my link is just different versions of the same file), and yes, if you programmatically want to look up words this is ideal. If you can guarantee the program will be executed only online, there are other options, but nevertheless I'm answering this part of the original question: "Alternatively, is there any way I can pull down the dictionary data that backs a Wiktionary?" Jun 19, 2012 at 20:18
-
22
To keep it really simple, extract the words from the dump like this:
bzcat pages-articles.xml.bz2 | grep '<title>[^[:space:][:punct:]]*</title>' | sed 's:.*<title>\(.*\)</title>.*:\1:' > words
-
-
-
That's great, thanks! If you want to get the words with a dash or space in it, you should use:
bzcat pages-articles.xml.bz2 | grep '<title>\(.*\)</title>' | sed 's:.*<title>\(.*\)</title>.*:\1:' > words
– nico_lrxFeb 22, 2022 at 18:02
If you are using Python, you can use WiktionaryParser by Suyash Behera.
You can install it by
pip install wiktionaryparser
Example usage:
from pprint import pprint
from wiktionaryparser import WiktionaryParser
parser = WiktionaryParser()
word = parser.fetch('test')
pprint(word)
another_word = parser.fetch('test', 'french')
pprint(another_word)
# features
parser.set_default_language('french')
parser.exclude_part_of_speech('noun')
parser.include_relation('alternative forms')
You could use the revisions API:
Or the parse API:
https://en.wiktionary.org/w/api.php?action=parse&page=test&prop=wikitext&formatversion=2
More examples are provided in the documentation.
-
You can also add
&format=json
to the urls to have a formatted response.– remborgJul 25, 2022 at 11:35
You might want to try JWKTL out. I just found out about it ;)
-
1The citation that you refer to is broken. Here is a link to the JWKTL page ukp.tu-darmstadt.de/software/jwktl. It's not really what I believe the OP is looking for though. Jan 14, 2013 at 14:41
-
The second link is (effectively) broken. It redirects to a genetic page, Welcome to the Ubiquitous Knowledge Processing (UKP) Lab!. Sep 12, 2021 at 8:45
-
The Wikipedia reference leads to Extracting lexical semantic knowledge from Wikipedia and Wiktionary and "...JWKTL (Java-based WiKTionary Library)...". Sep 12, 2021 at 8:53
As mentioned earlier, the problem with this approach is that Wiktionary provides the information about all the words of all the languages. So the approach to check if a page exists using Wikipedia API won't work because there're a lot of pages for non-English words. To overcome this, you need to parse each page to figure out if there's a section describing the English word. Parsing wikitext isn't a trivial task, though in your case it's not that bad. To cover almost all the cases you need to just check if the wikitext contains the English
heading. Depending on the programming language you use, you can find some tools to build an AST from wikitext. This will cover most of the cases, but not all of them because Wiktionary includes some common misspellings.
As an alternative, you could try using Lingua Robot or something similar. Lingua Robot parses the Wiktionary content and provides it as a REST API. A non-empty response means that the word exists. Please note that, as opposed to Wiktionary, the API itself doesn't include any misspellings (at least at the moment of writing this answer). Please also note that the Wiktionary contains not only the words, but multi-word expressions.
Here's a start to parsing etymology and pronunciation data:
function parsePronunciationLine(line) {
let val
let type
line.replace(/\{\{\s*a\s*\|UK\s*\}\}\s*\{\{IPA\|\/?([^\/\|]+)\/?\|lang=en\}\}/, (_, $1) => {
val = $1
type = 'uk'
})
line.replace(/\{\{\s*a\s*\|US\s*\}\}\s*\{\{IPA\|\/?([^\/\|]+)\/?\|lang=en\}\}/, (_, $1) => {
val = $1
type = 'us'
})
line.replace(/\{\{enPR|[^\}]+\}\},?\s*\{\{IPA\|\/?([^\/\|]+)\/?\|lang=en}}/, (_, $1) => {
val = $1
type = 'us'
})
line.replace(/\{\{a|GA\}\},?\s*\{\{IPA\|\/?([^\/\|]+)\/?\|lang=en}}/, (_, $1) => {
val = $1
type = 'ga'
})
line.replace(/\{\{a|GA\}\},?.+\{\{IPA\|\/?([^\/\|]+)\/?\|lang=en}}/, (_, $1) => {
val = $1
type = 'ga'
})
// {{a|GA}} {{IPA|/ˈhæpi/|lang=en}}
// * {{a|RP}} {{IPA|/pliːz/|lang=en}}
// * {{a|GA}} {{enPR|plēz}}, {{IPA|/pliz/|[pʰliz]|lang=en}}
if (!val)
return
return { val, type }
}
function parseEtymologyPiece(piece) {
let parts = piece.split('|')
parts.shift() // The first one is ignored.
let ls = []
if (langs[parts[0]]) {
ls.push(parts.shift())
}
if (langs[parts[0]]) {
ls.push(parts.shift())
}
let l = ls.pop()
let t = parts.shift()
return [ l, t ]
// {{inh|en|enm|poisoun}}
// {{m|enm|poyson}}
// {{der|en|la|pōtio|pōtio, pōtiōnis|t=drink, a draught, a poisonous draught, a potion}}
// {{m|la|pōtō|t=I drink}}
// {{der|en|enm|happy||fortunate, happy}}
// {{cog|is|heppinn||lucky}}
}
Here is a gist with it more fleshed out.
-
-
1