5

Is there an implemented way in either Grafana or Influx db, that could help detect outliers in a data set, and if possible remove, or at least avoid displaying them from a dashboard that uses this data set; I used percentiles, but that does not seem to hide substantial outliers.

Thanks.

2
  • 1
    As a best-practice note: if you are automatically discarding outliers, don't just throw them away. Outliers need to be analyzed too. robjhyndman.com/hyndsight/omitting-outliers has some wisdom on this: "The most famous case of that is the non-detection of the hole in the ozone layer by NASA. The way I was told the story was that outliers had been automatically filtered from the data obtained from Nimbus-7. ... scientists went back and found the problem could have been detected a decade earlier if automated outlier filtering had not been applied by NASA."
    – Jason
    Commented Jan 9, 2018 at 22:19
  • 1
    Having said that: a common practice is to create two CQs: one to store the data that is within +/- 3 stddev of the rolling mean, and one to store/highlight data outside this range.
    – Jason
    Commented Jan 9, 2018 at 22:21

3 Answers 3

7

Grafana is meant to display data, and although it might have some tools to achieve outlier detection and elimination it is not the right place to clean up data.

InfluxDB is meant to store data and is also not the thing to use to clean up data. However, influxQL has a bunch of mathematical functions that you could use when querying for data; you could downsample, get the mean, etc. You could have a look here to see if anything does what you need https://docs.influxdata.com/influxdb/v1.4/query_language/functions/

EDIT: InfluxQL is not to be confused with IFQL which also stands for Influx Query Language https://github.com/influxdata/ifql

EDIT2: IFQL is now known as Flux, InfluxData’s new functional data scripting language designed for querying, analyzing, and acting on data, and is a more flexible/powerful language than InfluxQL.

There's something called Continuous Queries in InfluxDB that periodically run and save the results in a specified measurement (possibly the same, either overwriting or adding a new field). This method could be used for cleaning up data, so that it is simpler to query and display later. https://docs.influxdata.com/influxdb/v1.4/query_language/continuous_queries/

The preferred method for data processing with InfluxDB would be to use Kapacitor which is part of the TICK stack (Telegraf, InfluxDB, Chornograf, Kapacitor) and integrates well with influxDB and can process data both real time or in batches and save it back in InfluxDB. But Kapacitor might be an overkill for simple things. Have a look at the mathematical functions for influxql first. https://www.influxdata.com/time-series-platform/kapacitor/

2

Moving average in Meta Queries Plugin for Grafana may help to smooth spikes in data if you don't want to use Kapacitor and/or CQ.

2

If you know beforehand the acceptable range of values for a specific metric you can specify a filter by value (field_key and field_value in InfluxDB terminology) in the WHERE part of the query.

This is also supported by Grafana in both the "text edit" and the "visual" modes. It's not obvious because Grafana only suggests tag names and values in the WHERE autocomplete dropdown menu, but you can also type field_key Enter, then choose an operator such as < and type the number X as the threshold above/below which all numbers will be filtered out as outliers.

For example, consider that you have a series reporting the latency of your internet connection and let's assume that anything above 200ms is an outlier or erroneous value from the sensor.

You can filter out the erroneous values with the following query. In "text edit" mode:

SELECT mean("value") FROM "ms" WHERE ("entity_id" = 'speedtest_ping' AND "value" < 200) AND time >= now() - 2d GROUP BY time(5m), "entity_id" fill(none)

In visual mode:

Grafana visual mode - filter values with WHERE clause

Here the field_key is "value" but you can filter by whatever field exists in your measurements.

I have noticed that for Grafana to preserve your filter when you switch between the visual and the text edit modes, you have to specify it within the parenthesis, e.g. like above: WHERE ("entity_id" = 'speedtest_ping' AND "value" < 200)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.