3

Im curious about the current libraries for Scala & Akka which would allow me to elegantly build a workflow pipeline.

In my case a workflow is just a DAG of operations so actors/Akka feels like a good fit.

My question is what's the best approach? There are Libs like reactive streams which allow really elegant composition of a pipeline but they seem very record focused.

My use case is a flow of operations passing messages between them. Future composition is nice but syntax becomes unwieldy after a while. Maybe there is something better with scalaz and shapeless.

What are the approaches and tools to building a DSL for pipelines of computation steps using message passing?

2
  • 2
    Can you explain what it means to be "record focused" and why this isn't appropriate for "passing messages"? What is the difference between a "record" and a "message" in your eyes?
    – Rex Kerr
    Nov 21, 2014 at 15:11
  • Sorry. My pipeline is very much like a chain of method calls. Where parameter passing is a message. The streaming side of Akka seems very focused on taking a stream of data. I.e. Log file. My case would be a log file with 1 line. While it would work. Is it efficient?
    – NightWolf
    Nov 22, 2014 at 0:46

2 Answers 2

3

While still in early development (pre 1.0 as of writing), you should have a look at akka-streams, which are exactly that - a way to describe a computation graph and then run it asynchronously.

3
  • Thanks. The streaming side of Akka seems very focused on taking a stream of data. I.e. Log file. My case would be a log file with 1 line. While it would work. Is it efficient?
    – NightWolf
    Nov 22, 2014 at 0:47
  • Any examples of streams using a future as a source?
    – NightWolf
    Nov 22, 2014 at 0:49
  • Source(Future(42)) :-) Nov 22, 2014 at 11:06
2

If your pipeline is very much like a chain of method calls, use a chain of method calls!

There's no point making the solution more complicated than it needs to be; if it's well-modelled by a chain of methods calls, just use that. (Or functions, which you can compose.)

If you need something slightly more complicated but you don't actually need any message-passing, you might want something like AsyncFP or Scala.Rx.

If you need a multi-core solution, but you have stretches that look like method calls, then have a chain of method calls inside one stop. You could use Akka streams for that without having to worry so much about the overhead to useful computation ratio.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.