630

How do I split a sentence and store each word in a list? For example, given a string like "these are words", how do I get a list like ["these", "are", "words"]?


To split on other delimiters, see Split a string by a delimiter in python.

To split into individual characters, see How do I split a string into a list of characters?.

1
  • 5
    As it is, you will be printing the full list of words for each word in the list. I think you meant to use print(word) as your last line.
    – tgray
    Apr 13, 2009 at 14:08

9 Answers 9

536

Given a string sentence, this stores each word in a list called words:

words = sentence.split()
1
  • 1
    This does not eliminate the special symbols such as commas and points
    – AbdelKh
    Feb 5 at 9:38
476

To split the string text on any consecutive runs of whitespace:

words = text.split()      

To split the string text on a custom delimiter such as ",":

words = text.split(",")   

The words variable will be a list and contain the words from text split on the delimiter.

0
92

Use str.split():

Return a list of the words in the string, using sep as the delimiter ... If sep is not specified or is None, a different splitting algorithm is applied: runs of consecutive whitespace are regarded as a single separator, and the result will contain no empty strings at the start or end if the string has leading or trailing whitespace.

>>> line = "a sentence with a few words"
>>> line.split()
['a', 'sentence', 'with', 'a', 'few', 'words']
0
60

Depending on what you plan to do with your sentence-as-a-list, you may want to look at the Natural Language Took Kit. It deals heavily with text processing and evaluation. You can also use it to solve your problem:

import nltk
words = nltk.word_tokenize(raw_sentence)

This has the added benefit of splitting out punctuation.

Example:

>>> import nltk
>>> s = "The fox's foot grazed the sleeping dog, waking it."
>>> words = nltk.word_tokenize(s)
>>> words
['The', 'fox', "'s", 'foot', 'grazed', 'the', 'sleeping', 'dog', ',', 
'waking', 'it', '.']

This allows you to filter out any punctuation you don't want and use only words.

Please note that the other solutions using string.split() are better if you don't plan on doing any complex manipulation of the sentence.

[Edited]

3
  • 6
    split() relies on white-space as the separator, so it will fail to separate hyphenated words--and long-dash separated phrases will fail to split too. And if the sentence contains any punctuation without spaces, those will fail to stick. For any real-world text parsing (like for this comment), your nltk suggestion is much better than split()`.
    – hobs
    Dec 14, 2011 at 13:10
  • 4
    Potentially useful, although I wouldn't characterise this as splitting into "words". By any plain English definition, ',' and "'s" are not words. Normally, if you wanted to split the sentence above into "words" in a punctuation-aware way, you'd want to strip out the comma and get "fox's" as a single word.
    – Mark Amery
    Jan 25, 2016 at 17:52
  • 1
    Python 2.7+ as of April 2016. Sep 20, 2016 at 20:57
37

How about this algorithm? Split text on whitespace, then trim punctuation. This carefully removes punctuation from the edge of words, without harming apostrophes inside words such as we're.

>>> text
"'Oh, you can't help that,' said the Cat: 'we're all mad here. I'm mad. You're mad.'"

>>> text.split()
["'Oh,", 'you', "can't", 'help', "that,'", 'said', 'the', 'Cat:', "'we're", 'all', 'mad', 'here.', "I'm", 'mad.', "You're", "mad.'"]

>>> import string
>>> [word.strip(string.punctuation) for word in text.split()]
['Oh', 'you', "can't", 'help', 'that', 'said', 'the', 'Cat', "we're", 'all', 'mad', 'here', "I'm", 'mad', "You're", 'mad']
2
  • 4
    Nice, but some English words truly contain trailing punctuation. For example, the trailing dots in e.g. and Mrs., and the trailing apostrophe in the possessive frogs' (as in frogs' legs) are part of the word, but will be stripped by this algorithm. Handling abbreviations correctly can be roughly achieved by detecting dot-separated initialisms plus using a dictionary of special cases (like Mr., Mrs.). Distinguishing possessive apostrophes from single quotes is dramatically harder, since it requires parsing the grammar of the sentence in which the word is contained.
    – Mark Amery
    Jan 29, 2016 at 0:02
  • 2
    @MarkAmery You're right. It's also since occurred to me that some punctuation marks—such as the em dash—can separate words without spaces. Sep 30, 2016 at 8:57
17

I want my python function to split a sentence (input) and store each word in a list

The str().split() method does this, it takes a string, splits it into a list:

>>> the_string = "this is a sentence"
>>> words = the_string.split(" ")
>>> print(words)
['this', 'is', 'a', 'sentence']
>>> type(words)
<type 'list'> # or <class 'list'> in Python 3.0
16

If you want all the chars of a word/sentence in a list, do this:

print(list("word"))
#  ['w', 'o', 'r', 'd']


print(list("some sentence"))
#  ['s', 'o', 'm', 'e', ' ', 's', 'e', 'n', 't', 'e', 'n', 'c', 'e']
1
15

shlex has a .split() function. It differs from str.split() in that it does not preserve quotes and treats a quoted phrase as a single word:

>>> import shlex
>>> shlex.split("sudo echo 'foo && bar'")
['sudo', 'echo', 'foo && bar']

NB: it works well for Unix-like command line strings. It doesn't work for natural-language processing.

1
  • 1
    Use with caution, especially for NLP. It will crash on single quote strings like "It's good." with ValueError: No closing quotation
    – Igor
    Aug 9, 2020 at 18:09
1

Split the words without without harming apostrophes inside words Please find the input_1 and input_2 Moore's law

def split_into_words(line):
    import re
    word_regex_improved = r"(\w[\w']*\w|\w)"
    word_matcher = re.compile(word_regex_improved)
    return word_matcher.findall(line)

#Example 1

input_1 = "computational power (see Moore's law) and "
split_into_words(input_1)

# output 
['computational', 'power', 'see', "Moore's", 'law', 'and']

#Example 2

input_2 = """Oh, you can't help that,' said the Cat: 'we're all mad here. I'm mad. You're mad."""

split_into_words(input_2)
#output
['Oh',
 'you',
 "can't",
 'help',
 'that',
 'said',
 'the',
 'Cat',
 "we're",
 'all',
 'mad',
 'here',
 "I'm",
 'mad',
 "You're",
 'mad']

Not the answer you're looking for? Browse other questions tagged or ask your own question.