21

I'm generating some statistics for some English-language text and I would like to skip uninteresting words such as "a" and "the".

  • Where can I find some lists of these uninteresting words?
  • Is a list of these words the same as a list of the most frequently used words in English?

update: these are apparently called "stop words" and not "skip words".

3
  • 6
    The title edit was totally legit, and most importantly, /made it an actual question/. Why would you roll that back? It seems that someone with 13.9k rep would be able to phrase a question as, you know, a question. Aug 11, 2009 at 12:09
  • How about the non-english stop words?
    – adib
    Jul 5, 2011 at 16:48
  • you can find list of stop words at toolspot.org/list-english-stop-words.php
    – Sunny
    Sep 3, 2013 at 3:33

6 Answers 6

21

The magic word to put into Google is "stop words". This turns up a reasonable-looking list.

MySQL also has a built-in list of stop words, but this is far too comprehensive to my tastes. For example, at our university library we had problems because "third" in "third world" was considered a stop word.

4
  • 3
    The nltk (Natural Language Toolkit, a python library) comes with a bunch of resources including a stopword corpus (Porter et al.), "2,400 stopwords for 11 languages". You can use the stopword list independent of the toolkit.
    – alexis
    Oct 26, 2012 at 20:53
  • How do I access this corpus of 2,400 stopwords in NLTK? Jul 7, 2015 at 12:07
  • nltk.org/nltk_data
    – Thomas
    Jul 8, 2015 at 12:35
  • 1
    The English stop-words in NLTK are tokenized. So instead of "shouldn't" it lists "shouldn"
    – gidim
    Nov 5, 2016 at 17:34
5

these are called stop words, check this sample

1
5

Depending on the subdomain of English you are working in, you may have/wish to compile your own stop word list. Some generic stop words could be meaningful in a domain. E.g. The word "are" could actually be an abbreviation/acronym in some domain. Conversely, you may want to ignore some domain specific words depending on your application which you may not want to ignore in the domain of general English. E.g. If you are analyzing a corpus of hospital reports, you may wish to ignore words like 'history' and 'symptoms' as they would be found in every report and may not be useful (from a plain vanilla inverted index perspective).

Otherwise, the lists returned by Google should be fine. The Porter Stemmer uses this and the Lucene seach engine implementation uses this.

4

Get statistics about word frequency in large txt corpora. Ignore all words with frequency > some number.

2
  • 13
    lol, this is just the work I'm trying to avoid! Aug 2, 2009 at 7:35
  • There may be words that he would want to skip (because they're syntactic sugar in the English language) that are nonetheless not as common as words that he would want to keep (because they're typical to the domain). I can't think of any brilliant examples offhand, though. Maybe "thou" and "one"?
    – jprete
    Aug 4, 2009 at 18:41
2

I think I used the stopword list for German from here when I built a search application with lucene.net a while ago. The site contains a list for English, too, and the lists on the site are apparaently the ones that the lucene project use as default, too.

2

Typically these words will appear in documents with the highest frequency. Assuming you have a global list of words:

{ Word Count }

With the list of words, if you ordered the words from the highest count to the lowest, you would have a graph (count (y axis) and word (x axis) that is the inverse log function. All of the stop words would be at the left, and the stopping point of the "stop words" would be at where the highest 1st derivative exists.

This solution is better than a dictionary attempt:

  • This solution is a universal approach that is not bound by language
  • This attempt learns what words are deemed to be "stop words"
  • This attempt will produce better results for collections that are very similar, and produce unique word listings for items in the collections
  • The stop words can be recalculated at a later time (with this there can be caching and a statistical determination that the stop words may have changed from when they were calculated)
  • This can also eliminate time based or informal words and names (such as slang, or if you had a bunch of documents that had a company name as a header)

The dictionary attempt is better:

  • The lookup time is much faster
  • The results are precached
  • Its simple
  • Some else came up with the stop words.

Not the answer you're looking for? Browse other questions tagged or ask your own question.