65

In java 8, what's the best way to check if a List contains any duplicate?

My idea was something like:

list.size() != list.stream().distinct().count()

Is it the best way?

1
  • 3
    if you are not interested in knowing what are those duplicates , then it's the best way !! May 5, 2015 at 13:04

6 Answers 6

72

Your code would need to iterate over all elements. If you want to make sure that there are no duplicates simple method like

public static <T> boolean areAllUnique(List<T> list){
    Set<T> set = new HashSet<>();

    for (T t: list){
        if (!set.add(t))
            return false;
    }
    
    return true;
}

would be more efficient since it can give you false immediately when first non-unique element would be found.

This method could also be rewritten using Stream#allMatch which also is short-circuit (returns false immediately for first element which doesn't fulfill provided condition)
(assuming non-parallel streams and thread-safe environment)

public static <T> boolean areAllUnique(List<T> list){
    Set<T> set = new HashSet<>();
    return list.stream().allMatch(t -> set.add(t));
}

which can be farther shortened as @Holger pointed out in comment

public static <T> boolean areAllUnique(List<T> list){
    return list.stream().allMatch(new HashSet<>()::add);
}
20
  • 46
    This can be even a one-liner: return list.stream().allMatch(new HashSet<>()::add);
    – Holger
    May 5, 2015 at 13:10
  • 8
    Seems a bit dangerous to use a predicate with side effects here. May 5, 2015 at 14:17
  • 10
    Both. The spec requires the predicate to be stateless. The usual caveat about running in parallel applies. ConcurrentHashMap might help. There may be other issues but I haven't had coffee yet. :-) May 5, 2015 at 14:24
  • 7
    @Stuart Marks: I think, for a one liner which spans the entire life cycle of the Stream, it’s acceptable to rely on the programmer to recognize whether that stream use is multi-threaded or not. Of course, if in doubt, using a ConcurrentMap might help. And a stateful predicate might not be the best thing, but is sometimes unavoidable, distinct tests being the common example. Maybe you remember this one ;^)
    – Holger
    May 5, 2015 at 17:36
  • 6
    @MubasharAhmad It is not that it is wrong, but new HashSet<>(list) will iterate over entire list, while .allMatch(new HashSet<>()::add) is short-circuit, it will return false at first occurrence of duplicate element.
    – Pshemo
    Nov 27, 2017 at 1:24
17

I used the following:
1. return list.size() == new HashSet<>(list).size();.

I'm not sure how it compares to:
2. return list.size() == list.stream().distinct().count();
and
3. return list.stream().sequential().allMatch(new HashSet<>()::add);
in terms of performance.

The last one (#3) has possibility to handle not only collections (e.g. lists), but also streams (without explicitly collecting them).

Upd.: The last one (#3) seems to be the best not only because it can handle pure streams, but also because it stops on the first duplicate (while #1 and #2 always iterate till the end) — as @Pshemo said in comment.

8

You can use the counting collector.

Stream.of(1, 3, 4, 6, 7, 5, 6)
            .collect(Collectors.groupingBy(
                    Function.identity(), Collectors.counting()))
            .entrySet().stream().anyMatch(e -> e.getValue() > 1)
2

Started this class as a StreamTool, but I think there must be an even better way with reduce or similar:

public class StreamTool {

    /**
     * Whether stream records are unique in that stream.
     * @param <T> Type of records
     * @param records
     * @return true if there are no duplicates, false otherwise
     */
    public static <T> boolean isUnique(Stream<T> records) {
        return records.allMatch(new HashSet<>()::add);
    }
}
2

Given array arr,

arr.length != Arrays.stream(arr).distinct().count()

will help check for duplicates

1
  • 4
    The question was not about arrays.
    – snieguu
    Nov 17, 2019 at 19:30
0

Use set.add() it is faster.

Set<T> items = new HashSet<>();
list.stream().filter(n -> !items.add(n)) 
            .collect(Collectors.toSet());
1
  • What is the point of creating two Set objects (one via new HashSet and other via Collections.toSet()? Also above code could be reduce to new HashSet<>(list), but question is not about removing duplicates, it is about checking if duplicate elements exist.
    – Pshemo
    Apr 22, 2022 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.