14

Using InfluxDB (v1.1), I have the requirement where I want to get the last entry timestamp for a specific key. Regardless of which measurement this is stored and regardless of which value this was.

The setup is simple, where I have three measurements: location, network and usage.
There is only one key: device_id.

In pseudo-code, this would be something like:

# notice the lack of a FROM clause on measurement here...
SELECT MAX(time) WHERE 'device_id' = 'x';

The question: What would be the most efficient way of querying this?

The reason why I want this is that there will be a decentralised sync process. Some devices may have been updated in the last hour, whilst others haven't been updated in months. Being able to get a distinct "last updated on" timestamp for a device (key) would allow me to more efficiently store new points to Influx.

I've also noticed there is a similar discussion on InfluxDB's GitHub repo (#5793), but the question there is not filtering by any field/key. And this is exactly what I want: getting the 'last' entry for a specific key.

3 Answers 3

15

Unfortunately there wont be single query that will get you what you're looking for. You'll have to do a bit of work client side.

The query that you'll want is

SELECT last(<field name>), time FROM <measurement> WHERE device_id = 'x'

You'll need to run this query for each measurement.

SELECT last(<field name>), time FROM location WHERE device_id = 'x'
SELECT last(<field name>), time FROM network WHERE device_id = 'x'
SELECT last(<field name>), time FROM usage WHERE device_id = 'x'

From there you'll get the one with the greatest time stamp

> select last(value), time from location where device_id = 'x'; select last(value), time from network where device_id = 'x'; select last(value),     time from usage where device_id = 'x';
name: location
time                last
----                ----
1483640697584904775 3

name: network
time                last
----                ----
1483640714335794796 4

name: usage
time                last
----                ----
1483640783941353064 4
2
  • Thanks Michael. Indeed, this is a solution to work with, so I've upvoted this. However, it will require three API calls - one for each measurement - to determine the value. That still is not really efficient. Also, more importantly, the queries you mention actually require specifying the <field_name>; this is something I do not know upfront. Therefor, I haven't marked it as an answer just yet. Perhaps future Influx versions will have a neat solution for this.
    – Juliën
    Feb 2, 2017 at 11:30
  • Makes sense. If you could open an issue on the InfluxDB repo asking for this kind of functionality, we can appropriately plan for it. Feb 6, 2017 at 20:47
14

tl;dr; The first() and last() selectors will NOT work consistently if the measurement have multiple fields, and fields have NULL values. The most efficient solution is to use these queries

First:

SELECT * FROM <measurement> [WHERE <tag>=value] LIMIT 1

Last:

SELECT * FROM <measurement> [WHERE <tag>=value] ORDER BY time DESC LIMIT 1

Explanation:

If you have a single field in your measurement, then the suggested solutions will work, but if you have more than one field and values can be NULL then first() and last() selectors won't work consistently and may return different timestamps for each field. For example, let's say that you have the following data set:

time                   fieldKey_1     fieldKey_2     device
------------------------------------------------------------
2019-09-16T00:00:01Z   NULL           A              1
2019-09-16T00:00:02Z   X              B              1
2019-09-16T00:00:03Z   Y              C              2
2019-09-16T00:00:04Z   Z              NULL           2

In this case querying

SELECT first(fieldKey_1) FROM <measurement> WHERE device = "1" 

will return

time                   fieldKey_1
---------------------------------
2019-09-16T00:00:02Z   X         

and the same query for first(fieldKey_2) will return a different time

time                   fieldKey_2
---------------------------------
2019-09-16T00:00:01Z   A

A similar problem will happen when querying with last.

And in case you are wondering, it wouldn't do querying 'first(*)' since you'll get an 'epoch-0' time in the results, such as:

 time                   first_fieldKey_1    first_fieldKey_2
 -------------------------------------------------------------
 1970-01-01T00:00:00Z   X                   A

So, the solution would be querying using combinations of LIMIT and ORDER BY. For instance, for the first time value you can use:

SELECT * FROM <measurement> [WHERE <tag>=value] LIMIT 1

and for the last one you can use

SELECT * FROM <measurement> [WHERE <tag>=value] ORDER BY time DESC LIMIT 1

It is safe and fast as it will relay on indexes.

Is curious to mention that this more simple approach was mentioned in the thread linked in the opening post, but was discarded. Maybe it was just lost overlooked.

Here there's a thread in InfluxData blogs about the subject also suggesting to use this approach.

3
  • I am currently facing the problem with empty field values. I have tried your solution but the select statements performing really poor. Did you test this on a larger set?
    – wero026
    Aug 3, 2021 at 8:40
  • @rw026, it was a dataset of ~4 million records with 15 fields, and the response time was almost immediate. It's worth to mention that the solution I proposed is a simple query based on ordering by time which is always indexed, shouldn't be a high impact query. I think there's a chance of short availability on computing resources in your server, I would check that first. Alternatively I would use some caching mechanism if possible, or prefetch this info to have it ready when needed. Aug 4, 2021 at 13:35
  • My issue was that I have more than 700 fields provided by many sensors. I think SELECT * is then no option anymore. Resolved it by adding the tag as field too and used first and last on it. Since this field is always present this solution works very well.
    – wero026
    Aug 4, 2021 at 14:59
6

I tried this and it worked for me in a single command :

SELECT last(<field name>), time FROM location, network, usage WHERE device_id = 'x'

The result I got :

name: location
time                last
----                ----
1483640697584904775 3

name: network
time                last
----                ----
1483640714335794796 4

name: usage
time                last
----                ----
1483640783941353064 4
2
  • Thanks. Not a bad idea and this would work, but only if the same <field name> exists in all three measurements, right? In my original question, it is not. But if for others this is the case, this oneliner is a nice addition.
    – Juliën
    Jan 11, 2019 at 15:33
  • Yes it only works if <field name> exists in all three measurements.
    – JasonG-FR
    Jan 11, 2019 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.