3

I'm working with pmap and Distributed arrays in Julia in order to parallelize a code. When the command @everywhere include("./code.jl") is run, a long list of warning messages is outputted, and in the process consumes a considerable amount of time. The packages that are inside the file code.jl are, for example:

using DataFrames
using Images

I'm not sure how to address the problem of missing from the cache

This is the line that is run

@everywhere include("./code.jl")

I did not expect a litany of complaints. However, the output is similar to this on all the workers:

From worker 2:  │ This may mean CategoricalArrays [324d7699-5711-5eae-9e2f-1d82baa6b597] does not support precompilation but is imported by a module that does.
      From worker 2:    └ @ Base loading.jl:947
      From worker 6:    ┌ Warning: Module CategoricalArrays with build ID 5344443537363826 is missing from the cache.
From worker 4:  ┌ Warning: Module DataFrames with build ID 5344463834994296 is missing from the cache.
      From worker 4:    │ This may mean DataFrames [a93c6f00-e57d-5684-b7b6-d8193f3e46c0] does not support precompilation but is imported by a module that does.
      From worker 4:    └ @ Base loading.jl:947
      From worker 2:    ┌ Warning: Module DataFrames with build ID 5344464953933087 is missing from the cache.
      From worker 2:    │ This may mean DataFrames [a93c6f00-e57d-5684-b7b6-d8193f3e46c0] does not support precompilation but is imported by a module that does.

2 Answers 2

6

You may have a package that needs recompilation, and you're getting a race condition in which all workers decide they need to independently recompile it. Try starting a single julia session and typing ]precompile (the ] for entering pkg mode). Once that finishes, for safety you could try using DataFrames just to make sure it worked. Then try your code above.

Sign up to request clarification or add additional context in comments.

2 Comments

Does this step need to be carried out every time the program is called?
No, only if packages have changed.
4

While @tholy answer is right - you will need to remember to recompile every time you update your packages.

Another approach to fight the race condition is to import the packages locally and than in distributed version:

using DataFrames
@everywhere using DataFrames

Assuming that your file only imports packages and defines functions you could try:

include("./code.jl")
@everywhere include("./code.jl")

This should help and you do not need to remember to recompile when you update some of the packages used by your application.

6 Comments

Your assumption that the file only imports packages (using ...) and defines functions is correct. I still get this long list of warning messages even after following your prescription.
is your code running on a single machine or distributed?
On a single node with multiprocessors
So two questions: (1) do you have this list when you run include("./code.jl") or only when @everywhere include("./code.jl") is run after include("./code.jl")? (2) is your code a module (if so you need to make sure it is compiled as well)
In that case it is not a problem with distributed computing but with your package installation. Try to delete the ~/.julia folder and reinstall packages
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.