57

I'm trying to learn python. Here is the relevant part of the exercise:

For each word, check to see if the word is already in a list. If the word is not in the list, add it to the list.

Here is what I've got.

fhand = open('romeo.txt')
output = []

for line in fhand:
    words = line.split()
    for word in words:
        if word is not output:
            output.append(word)

print sorted(output)

Here is what I get.

['Arise', 'But', 'It', 'Juliet', 'Who', 'already', 'and', 'and', 'and',
 'breaks', 'east', 'envious', 'fair', 'grief', 'is', 'is', 'is',
 'kill', 'light', 'moon', 'pale', 'sick', 'soft', 'sun', 'sun',
 'the', 'the', 'the', 'through', 'what', 'window', 'with', 'yonder']

Note duplication (and, is, sun, etc).

How do I get only unique values?

1
  • 5
    The idiomatic way is to maintain a set of words to check against. All those linear scans over a growing list makes an otherwise linear algorithm degrade to quadratic. Feb 19, 2017 at 23:31

5 Answers 5

97

To eliminate duplicates from a list, you can maintain an auxiliary list and check against.

myList = ['Arise', 'But', 'It', 'Juliet', 'Who', 'already', 'and', 'and', 'and', 
     'breaks', 'east', 'envious', 'fair', 'grief', 'is', 'is', 'is', 'kill', 'light', 
     'moon', 'pale', 'sick', 'soft', 'sun', 'sun', 'the', 'the', 'the', 
     'through', 'what', 'window', 'with', 'yonder']

auxiliaryList = []
for word in myList:
    if word not in auxiliaryList:
        auxiliaryList.append(word)

output:

['Arise', 'But', 'It', 'Juliet', 'Who', 'already', 'and', 'breaks', 'east', 
  'envious', 'fair', 'grief', 'is', 'kill', 'light', 'moon', 'pale', 'sick',
  'soft', 'sun', 'the', 'through', 'what', 'window', 'with', 'yonder']

This is very simple to comprehend and code is self explanatory. However, code simplicity comes on the expense of code efficiency as linear scans over a growing list makes a linear algorithm degrade to quadratic.


If the order is not important, you could use set()

A set object is an unordered collection of distinct hashable objects.

Hashability makes an object usable as a dictionary key and a set member, because these data structures use the hash value internally.

Since the average case for membership checking in a hash-table is O(1), using a set is more efficient.

auxiliaryList = list(set(myList))

output:

['and', 'envious', 'already', 'fair', 'is', 'through', 'pale', 'yonder', 
 'what', 'sun', 'Who', 'But', 'moon', 'window', 'sick', 'east', 'breaks', 
 'grief', 'with', 'light', 'It', 'Arise', 'kill', 'the', 'soft', 'Juliet']
3
  • hey I had the same problem , but here consider a list that grows exponentially almost daily , how do you handle that , I have done the same using a while and pop , but its really slow
    – dev
    Jan 12, 2021 at 14:48
  • @ETL_Devs better ask a new question with context to where are you getting the data from? are you writing it to disk at end of day? etc... Jan 12, 2021 at 15:35
  • With existing list l and new list n to add only uniques from I like to write it with the extend method like this, since it reads easy without a loop: l.extend([i for i in n if i not in l])
    – kontur
    Jun 9, 2021 at 14:22
27

Instead of is not operator, you should use not in operator to check whether the item is in the list:

if word not in output:

BTW, using set is a lot efficient (See Time complexity):

with open('romeo.txt') as fhand:
    output = set()
    for line in fhand:
        words = line.split()
        output.update(words)

UPDATE The set does not preserve the original order. To preserve the order, use the set as an auxiliary data structure:

output = []
seen = set()
with open('romeo.txt') as fhand:
    for line in fhand:
        words = line.split()
        for word in words:
            if word not in seen:  # faster than `word not in output`
                seen.add(word)
                output.append(word)
5
  • 1
    The exercise calls for a list where the words will be ordered according to the order of their first appearance, so I don't see how set() could replace the list, though it would clearly be a helpful auxiliary data structure. Feb 19, 2017 at 23:36
  • @JohnColeman, Thanks for the comment. I thought it doesn't matter, because OP used sorted at the end of the code. I will update the answer to include the version that preserve the order.
    – falsetru
    Feb 19, 2017 at 23:38
  • I see what you mean. I took that to be their way of checking for duplicates. The problem description itself didn't specify sorting the result. Feb 19, 2017 at 23:41
  • would it not be more efficient to build the set and convert it to a list at the end, outside of the loop? rather than appending repeatedly?
    – PatrickT
    Jun 14, 2020 at 6:28
  • 1
    @PatrickT, The first code does as you mentioned. As mentioned in the answer The set does not preserve the original order.; if you want the result result to keep order of appearance in the field, you should do it like the second code.
    – falsetru
    Jun 14, 2020 at 7:40
8

One method is to see if it's in the list prior to adding, which is what Tony's answer does. If you want to delete duplicate values after the list has been created, you can use set() to convert the existing list into a set of unique values, and then use list() to convert it into a list again. All in just one line:

list(set(output))

If you want to sort alphabetically, just add a sorted() to the above. Here's the result:

['Arise', 'But', 'It', 'Juliet', 'Who', 'already', 'and', 'breaks', 'east', 
 'envious', 'fair', 'grief', 'is', 'kill', 'light', 'moon', 'pale', 'sick', 
 'soft', 'sun', 'the', 'through', 'what', 'window', 'with', 'yonder']
1
  • 2
    If order doesn't matter then IMO this is the best answer so far as it will perform linearly and is also more concise. It does produces extra objects but that i okay for a lab/one-off program. This list -> set -> list pattern is used quite commonly
    – rsjethani
    Jun 27, 2018 at 8:03
6

Here's a "one-liner" which uses this implementation of removing duplicates while preserving order:

def unique(seq):
    seen = set()
    seen_add = seen.add
    return [x for x in seq if not (x in seen or seen_add(x))]

output = unique([word for line in fhand for word in line.split()])

The last line flattens fhand into a list of words, and then calls unique() on the resulting list.

0

fh = open('romeo.txt')
content = fh.read()
words = content.split()

mylist = list()
for word in words:
    if word not in mylist:
        mylist.append(word)

mylist.sort()
print(mylist)

fh.close()

4
  • Hi @7ud02, welcome on SO. Here are two hints, first is it easier for users to understand your answer when you and some prosa to your code. Seconde, the question was about to avoid duplicates in the list. And I think you answer doesn't solve this problem. Please think about updating your answer.
    – mosc9575
    Jan 12, 2021 at 15:58
  • First you read the content of the file, then create a list by using split(). Afterwards it adds only unique words to the newly created list(). In the end you sort and print the list. So there are NO duplicates at all.
    – 7ud02
    Jan 12, 2021 at 17:14
  • Your approche looks similar to the one in the question, so I was confused. Futhermore I want to tell you to never use keywords of the python language as a variable. This can cause some very difficult errors.
    – mosc9575
    Jan 12, 2021 at 18:16
  • 1
    Hello, Welcome to Stack Overflow! While this code may answer the question, providing additional context regarding why and/or how this code answers the question improves its long-term value.
    – ppwater
    Jan 13, 2021 at 1:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.