6

I heard that being conscious of type-stability contributes a lot to the high performance in Julia programming, so I tried to measure how much time I can save when rewriting the type-unstable function into type-stable version. As many people say, I assumed that type-stable coding of course has higher performance than type-unstable one. However, the result was otherwise:

# type-unstable vs type-stable

# type-unstable
function positive(x)
    if x < 0
        return 0.0
    else
        return x
    end
end

# type-stable
function positive_safe(x)
    if x < 0
        return zero(x)
    else
        return x
    end
end

@time for n in 1:100_000_000
    a = 2^( positive(-n) + 1 )
end

@time for n in 1:100_000_000
    b = 2^( positive_safe(-n) + 1 )
end

result:

0.040080 seconds
0.150596 seconds

I cannot believe this. Are there some mistakes in my code? Or this is the fact?

Any information would be appreciated.

Context

  • Operating System and version: Windows 10
  • Browser and version: Google Chrome 90.0.4430.212(Official Build) (64 bit)
  • JupyterLab version: 3.0.14

@btime result

Just replacing @time with @btime for my code above

@btime for n in 1:100_000_000
    a = 2^( positive(-n) + 1 )
end
# -> 1.500 ns 

@btime for n in 1:100_000_000
    b = 2^( positive_safe(-n) + 1 )
end
# -> 503.146 ms

Still weird.

the exact same code DNF showed me

using BenchmarkTools

@btime 2^(positive(-n) + 1) setup=(n=rand(1:10^8))
# -> 32.435 ns (0 allocations: 0 bytes)
@btime 2^(positive_safe(-n) + 1) setup=(n=rand(1:10^8))

#-> 3.103 ns (0 allocations: 0 bytes)

Works as expected.

I still don't understand what is happening. I feel like I have to know better about the usage of @btime and benchmarking process.

By the way, as I said above, I'm trying this benchmarking on Jupyterlab.

3
  • I'd say you're benchmarking the cost of the extra function call.
    – user14215102
    Jun 4, 2021 at 10:48
  • wiki.c2.com/?PrematureOptimization
    – user14215102
    Jun 4, 2021 at 10:51
  • 3
    @dratenik You don't know that this optimization is premature. I don't think that's a helpful comment to a reasonable question.
    – DNF
    Jun 4, 2021 at 11:03

3 Answers 3

6

The problem with your benchmark, you testing different logic code:

2 ^ (integer value) and 2 ^ (float value)

But the most crucial part, if a and b is not defined before the loop, Julia compiler may remove the block. Your performance very much depends was the a and b defined before and were defined in the global scope or not.

And power is the time-consuming central part of your code (not the type unstable part).

positive function returns Float in your case, positive_safe returns Int)

The code similar to your case (by logic) could look like that:

# type-unstable

function positive(x)
    if x < 0
        return 0.0
    else
        return x
    end
end

# type-stable
function positive_safe(x)
    if x < 0
        return 0.0
    else
        return Float64(x)
    end
end

function test1()
    a = 0.0
    for n in 1:100_000_000
        a += 2^( positive(-n) + 1 )
    end
    a
end

function test2()
    b = 0.0
    for n in 1:100_000_000
        b += 2^( positive_safe(-n) + 1 )
    end
    b
end

@btime test1()
@btime test2()  
98.045 ms (0 allocations: 0 bytes)
2.0e8

  97.948 ms (0 allocations: 0 bytes)
2.0e8

The results are almost the same since your type unstable is not a bottleneck for the case.

If to test the function (which is similar to your case when a/b was not defined):

function test3()
    b = 0.0
    for n in 1:100_000_000
        b += 2^( positive_safe(-n) + 1 )
    end
    nothing
end

@btime test3()

Benchmark will show results:

1.611 ns 

This is not because my laptop did 100_000_000 iterations per 1.611 ns, but because Julia compiler smart enough to understand that the test3 function may be replaced with nothing.

2

This is benchmarking problem. The @time macro is not suitable for microbenchmarks. Use the BenchmarkTools.jl package, and read the user manual. It is easy to make mistakes when benchmarking.

Here's how to do it:

jl> using BenchmarkTools

jl> @btime 2^(positive(-n) + 1) setup=(n=rand(1:10^8))
  6.507 ns (0 allocations: 0 bytes)
2.0

jl> @btime 2^(positive_safe(-n) + 1) setup=(n=rand(1:10^8))
  2.100 ns (0 allocations: 0 bytes)
2

As you see, the type stable function is faster.

15
  • 1
    @DNF Thank you for your research. However, even if I try @btime macro instead of @time, I still get the result which says positive is faster than positive_safe. 1.500 ns for positive, 503.146 ms for positive_safe.
    – ten
    Jun 4, 2021 at 11:40
  • 2
    @ForceBru The BenchmarkTools package takes care of compilation and warmup, chooses an appropriate number of executions, and calculates various statistics. It also allows you to interpolate variables to avoid issues with globals that can cause type instabilities. The @time macro does none of those things, at the very least, one must manually wrap code in functions, call the benchmark one initial time to avoid including compilation time, and use a loop to reduce statistical noise.
    – DNF
    Jun 4, 2021 at 12:00
  • 1
    @DNF I was trying @btime on my first code by just replacing @time with @btime. As I added the question, I ran the exact same code you showed me, and I saw positive_safe was faster than positive. Is it that I wrote wrong syntax for @btime?
    – ten
    Jun 4, 2021 at 12:09
  • 1
    @DNF, well, OP is already using a loop. Calling @time multiple times to mitigate the effects of compilation doesn't change the results in this case. Furthermore, replacing @time with @btime (and even @benchmark) in OP's code consistently shows that calling positive_safe in the loop takes around 290 ms, while calling positive takes a bit more than 2 nanoseconds, so positive is clearly much faster, according to this benchmark. What's the difference between OP's new benchmark (with the loop) and the one in your answer?
    – ForceBru
    Jun 4, 2021 at 12:09
  • 2
    @BenoitPasquier I don't really know. Apparently the compiler is able to constant fold the expression for the unsafe loop, perhaps the extra function call zero() is somehow preventing that in the second loop. But this shows how finicky microbenchmarks can be, and that one should read the performance tips, wrap code in functions and generally be careful with over-interpreting results.
    – DNF
    Jun 4, 2021 at 12:19
0

The problem, as Vitaliy said, is that powers in floating point done with logs can be faster than the integer ones that can be done as loop multiplies:

using BenchmarkTools
# type-unstable vs type-unstable

# type-unstable
function positive_float_unstable(x)
    if x < 0
        return 0.0
    else
        return x
    end
end

# type-unstable
function positive_int_unstable(x)
    if x < 0
        return 0
    else
        return x
    end
end

# type-stable
function positive_float_stable(x)
    if x < 0
        return 0.0
    else
        return Float64(x)
    end
end

# type-stable
function positive_int_stable(x)
    if x < 0
        return 0
    else
        return Int(x)
    end
end


println("unstable float")
@btime for n in 1:100_000_000
    a = 2^( positive_float_unstable(-n) + 1 )
end

println("unstable int")
@btime for n in 1:100_000_000
    b = 2^( positive_int_unstable(-n) + 1 )
end

println("stable float")
@btime for n in 1:100_000_000
    a = 2^( positive_float_stable(-n) + 1 )
end

println("stable int")
@btime for n in 1:100_000_000
    b = 2^( positive_int_stable(-n) + 1 )
end

Results:

unstable float
1.300 ns (0 allocations: 0 bytes) 

unstable int
179.232 ms (0 allocations: 0 bytes)

stable float
1.300 ns (0 allocations: 0 bytes)

stable int
178.990 ms (0 allocations: 0 bytes)

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.