1

I am currently attempting to implement a metaheuristic (genetic) algorithm. In this venture i also want to try and create somewhat fast and efficient code. However, my experience in creating efficient coding is not very great. I was therefore wondering if some people could give some "quick tips" to increase the efficiency of my code. I have created a small functional example of my code which contains most of the elements that the code will contain i regards to preallocating arrays, custom mutable structs, random numbers, pushing into arrays etc.

The options that I have already attempted to explore are options in regards to the package "StaticArrays". However many of my arrays must be mutable (so we need MArrays) and many of them will become very large > 100. The documentation of StaticArrays specify that the size of the StaticArrays package must remain small to remain efficient.

According to the documentation Julia 1.5.2 should be thread safe in regards to rand(). I have therefor attempted to multithread for-loops in my functions to make them run faster. And this results in a slight performance increase .

However if people can se a more efficient way of allocating Arrays or pushing in SpotPrices into an array it would be greatly appreciated! Any other performance tips are also very welcome!

# Packages
clearconsole()
using DataFrames
using Random
using BenchmarkTools
Random.seed!(42)

df = DataFrame( SpotPrice = convert(Array{Float64}, rand(-266:500,8832)),
month = repeat([1,2,3,4,5,6,7,8,9,10,11,12]; outer = 736),
hour = repeat([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24]; outer = 368))

# Data structure for the prices per hour
mutable struct SpotPrices
    hour :: Array{Float64,1}
end

# Fill-out data structure
function setup_prices(df::DataFrame)
    prices = []
    for i in 1:length(unique(df[:,3]))
        push!(prices, SpotPrices(filter(row -> row.hour == i, df).SpotPrice))
    end
    return prices
end

prices = setup_prices(df)

# Sampler function
function MC_Sampler(prices::Vector{Any}, sample_size::Int64)
    # Picking the samples
    tmp = zeros(sample_size, 24)

    # Sampling per hour
    for i in 1:24
        tmp[:,i] = rand(prices[i].hour, sample_size)
    end
    return tmp
end

samples = MC_Sampler(prices, 100)

@btime setup_prices(df)
@btime MC_Sampler(prices,100)

function setup_prices_par(df::DataFrame)
    prices = []
    @sync Threads.@threads for i in 1:length(unique(df[:,3]))
        push!(prices, SpotPrices(filter(row -> row.hour == i, df).SpotPrice))
    end
    return prices
end


# Sampler function
function MC_Sampler_par(prices::Vector{Any}, sample_size::Int64)
    # Picking the samples
    tmp = zeros(sample_size, 24)

    # Sampling per hour
    @sync Threads.@threads for i in 1:24
         tmp[:,i] = rand(prices[i].hour, sample_size)
    end
    return tmp
end

@btime setup_prices_par(df)
@btime MC_Sampler_par(prices,100)
1
  • Couple of minor comments on style: convert(Array{Float64}, rand(-266:500,8832)) should just be rand(-266.0:500.0, 8832). And repeat([1,2,3,4,5,6,7,8,9,10,11,12]; outer = 736) should just be repeat(1:12; outer=736).
    – DNF
    Commented Nov 15, 2020 at 19:30

1 Answer 1

4

Have a look at read very carefully https://docs.julialang.org/en/v1/manual/performance-tips/

Basic cleanups start with:

  1. Your SpotPrices struct does not need to me mutable. Anyway since there is only one field you could just define it as SpotPrices=Vector{Float64}
  2. You do not want untyped containers - instead of prices = [] do prices = Float64[]
  3. Using DataFrames.groupby will be much faster than finding unique elements and filtering by them
  4. If yo do not need initialze than do not do it Vector{Float64}(undef, sample_size) is much faster than zeros(sample_size, 24)
  5. You do not need to synchronize @sync before a multi-threaded loop
  6. Create a random states - one separate one for each thread and use them whenever calling the rand function
5
  • Brilliant! This was extremely helpful!
    – AliG
    Commented Nov 17, 2020 at 7:27
  • How much did performance increase after applying those recommendations? Commented Nov 17, 2020 at 12:15
  • The setup function went from 44.143 ms to 232.669 milliseconds. Quite impressive! And the MC_Sampler went form 55.565ms to 47.953ms. I had to add some functionality to the MC_Sampler function and I don't believe that I can now run it in parallel (they share a variable that changes every iteration). However the tips have also helped other bits of my code. A section went from 60ms down to 4 ms! I still have to create random states for each thread. Haven't gotten around to that yet.
    – AliG
    Commented Nov 19, 2020 at 7:01
  • sorry, 44.143 milliseconds to 232.669 microseconds
    – AliG
    Commented Nov 19, 2020 at 11:21
  • 1
    About random states - if you have problem with that just post a separate SO questions on how to assign random states to threads. Commented Nov 19, 2020 at 11:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.