2

I am trying to do a custom configuration apparently simple using Riemann and Collectd. Basically I'd like to calculate the ratio between two streams. In order to do that I tried something like (as in Rieamann API project suggestion here):

(project [(service "cahe-miss")
      (service "cache-all")]
  (smap folds/quotient
    (with :service "ratio"
    index)))

Which apparently works, but after a while I noticed some of the results where miss calculated. After log debugging I finished with the following configuration in order to see what's happening and proint the values:

(project [(service "cache-miss")
          (service "cache-all")]
  (fn [[miss all]]
    (if (or (nil? miss) (nil? all)) 
      (do nil) 
      (do (where (= (:time miss) (:time all))
        ;to print time marks
        (println (:time all)) 
        (println (:time miss))
        ; to distinguish easily each event
        (println "NEW LINE") 
      )) 
    )
  )
)

My surprise is that each time I get new data from collectd (every 10 seconds) the function I created is executed twice, like reusing previous unused data, and more over it looks like it doesn't care at all about my time equality constraint in the (where (= :time....) clasue. The problem is than I am dividing metrics with different time stamp. Below some ouput of the previous code:

1445606294
1445606294
NEW LINE -- First time I get data
1445606304
1445606294
NEW LINE
1445606304
1445606304
NEW LINE -- Second time I get data
1445606314
1445606304
NEW LINE
1445606314
1445606314
NEW LINE -- Third time I get data

Is there anyone that can give a hint on how to get the data formatted as I expected? I assume there is something I am not understading about the "project" function. Or something related to how incoming data is processed in riemann.

Thanks in advance!

Updated

I managed to solve my problem but still I don't have a clear idea of how it works, however I managed to do so. Right now I am receiving two different streams from collectd tail plugin (from nginx logs) and I managed to make the quotient between them as it follows:

(where (or (service "nginx/counter-cacheHit") (service "nginx/counter-cacheAll"))
    (coalesce
        (smap folds/quotient (with :service "cacheHit" (scale (* 1 100) index)))))

I have tested it widely and up to now it produces the right results. However I still don't understand several things... First, how it is that coalesce only returns data after both events are processed. Collectd sends the events of the both streams every two seconds with the same time mark, usin "project" instead of "coalesce" resulted in two different executions of smap each two seconds (one for each event), however coalesce results only with one execution of smap with the two events with the same time mark, which is exactly what I wanted.

Finally, I don't know which is the criteria to choose which is the numerator and denominator. Is it becaouse of the "or" clauses in "where" clause?

Anyway, with some blackmagic behind it but I managed to solve my problem ;^)

Thank you all!

1 Answer 1

0

taking the ratios between streams that where moving at different rates didn't work out for me. I have since settled on calculating ratios and rates within a fixed time interval or a moving time interval. This way you are capturing a consistent snapshot of events in a time block and calculating over this. Here is some elided code from comparing the rate a service is receiving events to the rate at which it is forwarding events:

(moving-time-window 30 ;; seconds
  (smap (fn [events]
          (let [in (or (->> events
                            (filter #(= (:service %) "event-received"))
                            count)
                       0)
                out (or (->> events
                             (filter #(= (:service %) "event-sent"))
                             count)
                        0)
                flow-rate (float (if (> in 0) (/ out in) 0))]
            {:service "flow rate"
             :metric flow-rate
             :host "All"
             :state (if (< flow-rate 0.99) "WARNING" "OK")
             :time (:time (last events))
             :ttl default-interval}))
        (tag ["some" "tags" "here"] index)
        (where (and
                (< (:metric event) 0.9)
                (= (:environment event) "production"))
               (throttle 1 3600 send-to-slack))))

This takes in a fixed window of events, calculates the ratio for that block and emits an event containing that ratio as it's metric. then if the metric is bad it calls me on slack.

1
  • Hi arthur, Sorry I could not comment before. I could not manage to run your suggestion, I get an exception, I will have to look it closer, however this is not really functional for me as you are counting events, and making a ratio with that counter. My situation is slightly different. I am trying to make a ratio within the metrics of two stream, that have exaclty the same time marks. I am trying to make the quotient of two different streams comming form the same plugin of collectd (tails plugin in that case). Thank so far, your option enlighted me.
    – Deivit
    Oct 29, 2015 at 8:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.