7

Is there a way to control the thread pools which handle the functions which get sent to agents? As I understand things, if I send-off, underneath the hood I'm using an unbounded thread pool. I would like to, say, run some functions on one thread pool and other functions on another. The reason for this is say I have a some functions which do IO and which are also less important. I'd throw these on some bounded thread pool and wouldn't worry if there was excessive blocking and they stacked up since they're, well, less important. The main thing is that I wouldn't want their crappy IO blocking to say have an effect on some more important functions which are running on another thread pool.

I'm basing the question off of something similar I did with thread pools in Akka and I'm just wondering I can accomplish the same thing with Clojure.

1
  • I would suggest using async IO to solve such problem rather than managing a thread pool for IO operations
    – Ankur
    Jul 4, 2012 at 4:12

3 Answers 3

14

For Clojure versions up to 1.4:

You cannot replace the built-in agent send and send-off thread pools. They are hard-coded in Agent.java.

The send pool (used for computation) is fixed size = 2 + Runtime.getRuntime().availableProcessors().

The send-off pool (also used for futures) is a cached thread pool and will grow without bound. This allows an arbitrary number of background tasks to wait for I/O. The cached threads will be reused and discarded if they've been idle for one minute.

If you want to manage work on your own thread pool, you'll need to dip into java.util.concurrent or use a Clojure wrapper lib.

For Clojure 1.5 (upcoming):

You can supply your own ExecutorService using (send-via executor a f), the default threadpools are not hard-wired anymore. See Agent.java in Clojure 1.5+ for details.

2
  • Any way to do the same defining of custom ThreadPools for Futures in 1.5+ ?
    – Didier A.
    Oct 23, 2015 at 22:07
  • @DidierA. based on the information in the answer (at the time of this reading) it looks like future uses the same Executor as send-off so I’d think we could set the executor used by future via set-agent-send-off-executor!
    – Avi Flax
    Apr 28, 2020 at 18:50
2

The Clojure library Claypoole is designed for exactly this. It lets you define threadpools and use (and reuse) them for futures, pmaps, and so on.

1
  • 1
    Correct me if I'm wrong, but I don't think Claypool gives you control over agent thread pools, only thread pools used for it's custom future, pmap and so on implementations. Aug 26, 2014 at 18:47
1

Amit Rathore (of Runa inc), has published a library (called medusa) for managing thread pools. It looks like a fairly close match for what you are looking for.

http://s-expressions.com/2010/06/08/medusa-0-1-a-supervised-thread-pool-for-clojure-futures-2/

2
  • is that integrated with agents, or is it separate? in other words - can you use it to replace the built-in thread pools? or is there some way to create agents with a different thread pool? i can't see anything that explains this... when i was worried about this i asked this q - stackoverflow.com/questions/10969708/parallel-doseq-for-clojure - and i don't think any answer gave a seamless way to replace the system functionality. Jul 4, 2012 at 1:46
  • medusa is an alternate library for running asynchronous tasks, it is an alternative to the built in agents. Jul 4, 2012 at 5:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.