11

I have a vector of 2500 values composed of repeated values and NaN values. I want to remove all the NaN values and compute the number of occurrences of each other value.

y
2500-element Array{Int64,1}:
8
43
NaN
46
NaN
8
8
3
46
NaN

For example: the number of occurences of 8 is 3 the number of occurences of 46 is 2 the number of occurences of 43 is 1.

1

3 Answers 3

16

To remove the NaN values you can use the filter function. From the Julia docs:

filter(function, collection)

Return a copy of collection, removing elements for which function is false.

x = filter(y->!isnan(y),y)
filter!(y->!isnan(y),y)

Thus, we create as our function the conditional !isnan(y) and use it to filter the array y (note, we could also have written filter(z->!isnan(z),y) using z or any other variable we chose, since the first argument of filter is just defining an inline function). Note, we can either then save this as a new object or use the modify in place version, signaled by the ! in order to simply modify the existing object y

Then, either before or after this, depending on whether we want to include the NaNs in our count, we can use the countmap() function from StatsBase. From the Julia docs:

countmap(x)

Return a dictionary mapping each unique value in x to its number of occurrences.

using StatsBase
a = countmap(y)

you can then access specific elements of this dictionary, e.g. a[-1] will tell you how many occurrences there are of -1

Or, if you wanted to then convert that dictionary to an Array, you could use:

b = hcat([[key, val] for (key, val) in a]...)'

Note: Thanks to @JeffBezanon for comments on correct method for filtering NaN values.

Sign up to request clarification or add additional context in comments.

2 Comments

Filtering with y->y!=NaN doesn't work, because NaN!=NaN is true (as per the rules of IEEE floating point arithmetic). Instead you can filter with y->!isnan(y).
@JeffBezanson Good catch, thanks! I corrected it in the response.
10
y=rand(1:10,20)
u=unique(y)
d=Dict([(i,count(x->x==i,y)) for i in u])
println("count for 10 is $(d[10])")

4 Comments

it works. but l can't access to the array value by value, for instance (6,1) how can l read only then only 1 ?. l need that to draw a histogram. x-axis represents the different values and the y-axis is the number of occurnces of each value .[(i,count(x->x==i,y)) for i in u] 9-element Array{Tuple{Any,Int64},1}: (6,1) (1,2) (7,3) (10,3) (9,3) (2,3) (5,1) (3,2) (8,2)
This works, and is very elegant. But it passes over the array y many times. If you have many unique values in y, it becomes unbearably slow, easily several orders of magnitude slower than necessary. countmap avoids this problem.
Not ruling out your comment (I actually didn't know about countmap and I'm checking out StatsBase), however, as you say so: "if you have". I'm convinced that premature optimization is evil and I'm a believer of "the rules of optimization"
Well, that's fair enough as a personal philosophy. I don't share the (apparently widespread) negative opinion of optimization. Furthermore, answers on Stackoverflow are visible for posterity, so I think that pointing out potential performance problems is a reasonable thing to do. Not all optimization is "premature", especially if it just involves applying general coding principles (e.g. "pass over an array just once.")
7

countmap is the best solution I've seen so far, but here's a written out version, which is only slightly slower. It only passes over the array once, so if you have many unique values, it is very efficient:

function countmemb1(y)
    d = Dict{Int, Int}()
    for val in y
        if isnan(val)
            continue
        end
        if val in keys(d)
            d[val] += 1
        else
            d[val] = 1
        end
    end
    return d
end

The solution in the accepted answer can be a bit faster if there are a very small number of unique values, but otherwise scales poorly.

Edit: Because I just couldn't leave well enough alone, here's a version that is more generic and also faster (countmap doesn't accept strings, sets or tuples, for example):

function countmemb(itr)
    d = Dict{eltype(itr), Int}()
    for val in itr
        if isa(val, Number) && isnan(val)
            continue
        end
        d[val] = get(d, val, 0) + 1
    end
    return d
end

2 Comments

Nice answer :) Why get! rather than get?
It's a pretty old answer, so I don't remember why I did this. You're right anyway, get is actually faster. get! will store the key in the dict if it's not already in there, but that's not needed in this case. I edited my answer, thanks for the tip!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.