120

I have tried reading up on this but I still don't understand the value of them or what they replace. And do they make my code shorter, more understandable or what?

Update

A lot of people posted answers, but it would be nice to see examples of with and without transducers for something very simple, which even an idiot like me can understand. Unless of course transducers need a certain high level of understanding, in which case I will never understand them :(

0

12 Answers 12

82

Transducers are recipes of what to do with a sequence of data without knowledge of what the underlying sequence is (how to do it). It can be any seq, async channel or maybe observable.

They are composable and polymorphic.

The benefit is, you don't have to implement all standard combinators every time a new data source is added. Again and again. As a result, you as user are able to reuse those recipes on different data sources.

Prior to version 1.7 of Clojure you had three ways to write dataflow queries:

  1. nested calls

    (reduce + (filter odd? (map #(+ 2 %) (range 0 10))))
    
  2. functional composition

    (def xform
      (comp
        (partial filter odd?)
        (partial map #(+ 2 %))))
    (reduce + (xform (range 0 10)))
    
  3. threading macro

    (defn xform [xs]
      (->> xs
           (map #(+ 2 %))
           (filter odd?)))
    (reduce + (xform (range 0 10)))
    

With transducers you will write it like:

(def xform
  (comp
    (map #(+ 2 %))
    (filter odd?)))
(transduce xform + (range 0 10))

They all do the same. The difference is that you never call transducers directly, you pass them to another function. Transducers know what to do, the function that gets a transducer knows how. The order of combinators is like you write it with threading macro (natural order). Now you can reuse xform with channel:

(chan 1 xform)
13
  • 4
    I was more looking for an answer which comes with an example which shows me how transducers save me time.
    – yazz.com
    Commented Oct 13, 2014 at 6:54
  • 6
    Its not a technical decision. We only use business value based decisions. "Just use them" will get me fired
    – yazz.com
    Commented Oct 21, 2014 at 8:21
  • 1
    You may have an easier time keeping your job if you delay trying to use transducers until Clojure 1.7 is released.
    – user100464
    Commented Oct 21, 2014 at 13:19
  • 10
    Transducers seem to be a useful way of abstracting over various forms of iterable objects. These can be non-consumable, such as Clojure seqs, or consumable (such as async channels). In this respect, it seems to me you would benefit greatly from using transducers if, e.g., you switch from a seq-based implementation to a core.async implementation using channels. Transducers should allow you to keep the core of your logic unchanged. Using traditional sequence-based processing you would have to convert this to use either transducers or some core-async analog. That's the business case. Commented Feb 23, 2015 at 18:56
  • 1
    I want to know whether exists the theory "Transducer" base on. Is it a common definition in functional languages or invented by the author of clojure ? It seems it treat questions as something like "chan","data flow" etc Commented Jul 2, 2015 at 12:37
54
+500

Transducers improve efficiency, and allow you to write efficient code in a more modular way.

This is a decent run through.

Compared to composing calls to the old map, filter, reduce etc. you get better performance because you don't need to build intermediate collections between each step, and repeatedly walk those collections.

Compared to reducers, or manually composing all your operations into a single expression, you get easier to use abstractions, better modularity and reuse of processing functions.

5
  • 2
    Just curious, you said above: "to build intermediate collections between each step". But doesn't "intermediate collections" sound like an anti-pattern? .NET offers lazy enumerables, Java offers lazy streams or Guava-driven iterables, lazy Haskell must have something lazy too. None of these requires map/reduce to use intermediate collections because all of them build an iterator chain. Where am I wrong here? Commented Oct 11, 2014 at 21:01
  • 3
    Clojure map and filter create intermediate collections when nested.
    – noisesmith
    Commented Oct 11, 2014 at 22:59
  • 4
    And at least regarding Clojure's version of laziness, the issue of laziness is orthogonal here. Yes, map and filter are lazy, the also generate containers for lazy values when you chain them. If you don't hold onto the head, you don't build up large lazy sequences that aren't needed, but you still build those intermediate abstractions for each lazy element.
    – noisesmith
    Commented Oct 12, 2014 at 0:59
  • An example woud be nice.
    – yazz.com
    Commented Oct 13, 2014 at 6:56
  • 9
    @LyubomyrShaydariv By "intermediate collection", noisesmith doesn't mean "iterate/reify an entire collection, then iterate/reify another entire collection". He or she means that when you nest function calls that return sequentials, each function call results in the creation of a new sequential. The actual iteration still only happens once, but there is additional memory consumption and object allocation due to the nested sequentials.
    – erikprice
    Commented Oct 30, 2014 at 15:58
29

Say you want to use a series of functions to transform a stream of data. The Unix shell lets you do this kind of thing with the pipe operator, e.g.

cat /etc/passwd | tr '[:lower:]' '[:upper:]' | cut -d: -f1| grep R| wc -l

(The above command counts the number of users with the letter r in either upper- or lowercase in their username). This is implemented as a set of processes, each of which reads from the previous processes's output, so there are four intermediate streams. You could imagine a different implementation that composes the five commands into a single aggregate command, which would read from its input and write its output exactly once. If intermediate streams were expensive, and composition were cheap, that might be a good trade-off.

The same kind of thing holds for Clojure. There are multiple ways to express a pipeline of transformations, but depending on how you do it, you can end up with intermediate streams passing from one function to the next. If you have a lot of data, it's faster to compose those functions into a single function. Transducers make it easy to do that. An earlier Clojure innovation, reducers, let you do that too, but with some restrictions. Transducers remove some of those restrictions.

So to answer your question, transducers won't necessarily make your code shorter or more understandable, but your code probably won't be longer or less understandable either, and if you're working with a lot of data, transducers can make your code faster.

This is a pretty good overview of transducers.

4
  • 1
    Ah, so transducers are mostly a performance optimisation, is that what you are saying?
    – yazz.com
    Commented Oct 13, 2014 at 9:10
  • @Zubair Yes, that's right. Note that the optimization goes beyond eliminating intermediate streams; you may also be able to perform operations in parallel.
    – user100464
    Commented Oct 14, 2014 at 19:17
  • 2
    It's worth mentioning pmap, which doesn't seem to get enough attention. If you are mapping an expensive function over a sequence, making the operation parallel is as easy as adding "p". No need to change anything else in your code, and it's available now--not alpha, not beta. (If the function creates intermediate sequences, then transducers might be faster, I would guess.)
    – Mars
    Commented Oct 30, 2014 at 15:13
  • 1
    Parallel is good for maximising core use but if you are dealing with large / infinite data then you will run out of memory, and hence lazy/streaming will help
    – HaveAGuess
    Commented Dec 12, 2021 at 17:28
25

Transducers are a means of combination for reducing functions.

Example: Reducing functions are functions that take two arguments: A result so far and an input. They return a new result (so far). For example +: With two arguments, you can think of the first as the result so far and the second as the input.

A transducer could now take the + function and make it a twice-plus function (doubles every input before adding it). This is how that transducer would look like (in most basic terms):

(defn double
  [rfn]
  (fn [r i] 
    (rfn r (* 2 i))))

For illustration substitute rfn with + to see how + is transformed into twice-plus:

(def twice-plus ;; result of (double +)
  (fn [r i] 
    (+ r (* 2 i))))

(twice-plus 1 2)  ;-> 5
(= (twice-plus 1 2) ((double +) 1 2)) ;-> true

So

(reduce (double +) 0 [1 2 3]) 

would now yield 12.

Reducing functions returned by transducers are independent of how the result is accumulated because they accumulate with the reducing function passed to them, unknowingly how. Here we use conj instead of +. Conj takes a collection and a value and returns a new collection with that value appended.

(reduce (double conj) [] [1 2 3]) 

would yield [2 4 6]

They are also independent of what kind of source the input is.

Multiple transducers can be chained as a (chainable) recipe to transform reducing functions.

Update: Since there now is an official page about it, I highly recommend to read it: http://clojure.org/transducers

8
  • 1
    Nice explanation but soon got into too much jargon for me, "Reducing functions generated by transducers are independent of how the result is accumulated".
    – yazz.com
    Commented Oct 13, 2014 at 6:57
  • 1
    You are right, the word generated was inappropriate here. Commented Oct 26, 2014 at 2:16
  • Its, ok. Anyway I understand that Transformers are just an optimisation now, so probably shouldn't be used anyway
    – yazz.com
    Commented Oct 27, 2014 at 8:04
  • 1
    They are a means of combination for reducing functions. Where else do you have that? This is far more than an optimization. Commented Oct 27, 2014 at 10:05
  • I find this answer very interesting, but it's not clear to me how it connects to transducers (partly because I still find the subject confusing). What is the relationship between double and transduce?
    – Mars
    Commented Apr 30, 2015 at 14:31
12

Rich Hickey gave a 'Transducers' talk at the Strange Loop 2014 conference (45 min).

He explains in simple way what transducers are, with real world examples - processing bags in an airport. He clearly separates the different aspects and contrasts them with the current approaches. Towards the end, he gives the rationale for their existence.

Video: https://www.youtube.com/watch?v=6mTbuzafcII

8

I've found reading examples from transducers-js helps me understand them in concrete terms of how I might use them in day-to-day code.

For instance, consider this example (taken from the README at the link above):

var t = require("transducers-js");

var map    = t.map,
    filter = t.filter,
    comp   = t.comp,
    into   = t.into;

var inc    = function(n) { return n + 1; };
var isEven = function(n) { return n % 2 == 0; };
var xf     = comp(map(inc), filter(isEven));

console.log(into([], xf, [0,1,2,3,4])); // [2,4]

For one, using xf looks much cleaner than the usual alternative with Underscore.

_.filter(_.map([0, 1, 2, 3, 4], inc), isEven);
2
  • How come the transducers example is so much longer. The underscore version looks a lot more concise
    – yazz.com
    Commented Oct 13, 2014 at 6:55
  • 1
    @Zubair Not really t.into([], t.comp(t.map(inc), t.filter(isEven)), [0,1,2,3,4]) Commented Nov 10, 2016 at 23:40
8

Transducers are (to my understanding!) functions which take one reducing function and return another. A reducing function is one which

For example:

user> (def my-transducer (comp count filter))
#'user/my-transducer
user> (my-transducer even? [0 1 2 3 4 5 6])
4
user> (my-transducer #(< 3 %) [0 1 2 3 4 5 6])
3

In this case my-transducer takes an input filtering function which it applies to 0 then if that value is even? in the first case the filter passes that value to the counter, then it filters the next value. Instead of first filtering and then passing all of those values over to count.

It is the same thing in the second example it checks one value at a time and if that value is less than 3 then it lets count add 1.

1
  • 1
    I liked this simple explanation
    – Ignacio
    Commented Aug 6, 2017 at 16:22
7

A transducer clear definition is here:

Transducers are a powerful and composable way to build algorithmic transformations that you can reuse in many contexts, and they’re coming to Clojure core and core.async.

To understand it, let's consider the following simple example:

;; The Families in the Village

(def village
  [{:home :north :family "smith" :name "sue" :age 37 :sex :f :role :parent}
   {:home :north :family "smith" :name "stan" :age 35 :sex :m :role :parent}
   {:home :north :family "smith" :name "simon" :age 7 :sex :m :role :child}
   {:home :north :family "smith" :name "sadie" :age 5 :sex :f :role :child}

   {:home :south :family "jones" :name "jill" :age 45 :sex :f :role :parent}
   {:home :south :family "jones" :name "jeff" :age 45 :sex :m :role :parent}
   {:home :south :family "jones" :name "jackie" :age 19 :sex :f :role :child}
   {:home :south :family "jones" :name "jason" :age 16 :sex :f :role :child}
   {:home :south :family "jones" :name "june" :age 14 :sex :f :role :child}

   {:home :west :family "brown" :name "billie" :age 55 :sex :f :role :parent}
   {:home :west :family "brown" :name "brian" :age 23 :sex :m :role :child}
   {:home :west :family "brown" :name "bettie" :age 29 :sex :f :role :child}

   {:home :east :family "williams" :name "walter" :age 23 :sex :m :role :parent}
   {:home :east :family "williams" :name "wanda" :age 3 :sex :f :role :child}])

What about it we want to know how many children are in the village? We can easily find it out with the following reducer:

;; Example 1a - using a reducer to add up all the mapped values

(def ex1a-map-children-to-value-1 (r/map #(if (= :child (:role %)) 1 0)))

(r/reduce + 0 (ex1a-map-children-to-value-1 village))
;;=>
8

Here is another way to do it:

;; Example 1b - using a transducer to add up all the mapped values

;; create the transducers using the new arity for map that
;; takes just the function, no collection

(def ex1b-map-children-to-value-1 (map #(if (= :child (:role %)) 1 0)))

;; now use transduce (c.f r/reduce) with the transducer to get the answer 
(transduce ex1b-map-children-to-value-1 + 0 village)
;;=>
8

Besides, it is really powerful when taking subgroups in account as well. For instance, if we would like to know how many children are in Brown Family, we can execute:

;; Example 2a - using a reducer to count the children in the Brown family

;; create the reducer to select members of the Brown family
(def ex2a-select-brown-family (r/filter #(= "brown" (string/lower-case (:family %)))))

;; compose a composite function to select the Brown family and map children to 1
(def ex2a-count-brown-family-children (comp ex1a-map-children-to-value-1 ex2a-select-brown-family))

;; reduce to add up all the Brown children
(r/reduce + 0 (ex2a-count-brown-family-children village))
;;=>
2

I hope you can find helpful these examples. You can find more here

Hope it helps.

Clemencio Morales Lucas.

3
  • 3
    The "Transducers are a powerful and composable way to build algorithmic transformations that you can reuse in many contexts, and they’re coming to Clojure core and core.async." definition could apply to almost anything?
    – yazz.com
    Commented Oct 16, 2014 at 11:10
  • 1
    To almost any Clojure Transducer, I would say. Commented Oct 16, 2014 at 11:20
  • 6
    It's more of a mission statement than a definition.
    – Mars
    Commented Oct 30, 2014 at 15:16
4

I blogged about this with a clojurescript example which explains how the sequence functions are now extensible by being able to replace the reducing function.

This is the point of transducers as I read it. If you think about the cons or conj operation that is hard coded in operations like map, filter etc., the reducing function was unreachable.

With transducers, the reducing function is decoupled and I can replace it like I did with the native javascript array push thanks to transducers.

(transduce (filter #(not (.hasOwnProperty prevChildMapping %))) (.-push #js[]) #js [] nextKeys)

filter and friends have a new 1 arity operation that will return a transducing function that you can use to supply your own reducing function.

4

Here's my (mostly) jargon and code free answer.

Think of data in two ways, a stream (values that occur over time such as events) or a structure (data that exists at a point in time such as a list, a vector, an array etc).

There are certain operations that you might want to perform over either streams or structures. One such operation is mapping. A mapping function might increment each data item (assuming it is a number) by 1 and you can hopefully imagine how this could apply to either a stream or a structure.

A mapping function is just one of a class of functions that are sometimes referred to as "reducing functions". Another common reducing function is filter which remove values that match a predicate (e.g. remove all values that are even).

Transducers let you "wrap" a sequence of one or more reducing functions and produce a "package" (which is itself a function) that works on both streams or structures. For example, you could "package" a sequence of reducing functions (e.g. filter even numbers, then map the resulting numbers to increment them by 1) and then use that transducer "package" on either a stream or structure of values (or both).

So what is special about this? Typically, reducing functions not able to be efficiently composed to work on both streams and structures.

So the benefit to you is that you can leverage your knowledge around these functions and apply them to more use cases. The cost to you is that you have to learn some extra machinery (i.e. the transducer) to give you this extra power.

3

As far as I understand, they're like building blocks, decoupled from the input and output implementation. You just define the operation.

As the implementation of the operation is not in the input's code and nothing is done with the output, transducers are extremely reusable. They remind me of Flows in Akka Streams.

I'm also new to transducers, sorry for the possibly-unclear answer.

1

I find this post give you an more bird eye view of transducer.

https://medium.com/@roman01la/understanding-transducers-in-javascript-3500d3bd9624

3

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.