5

I have a dataset in dynamoDB which looks like this:

{
    "userID" : 2323423,    // Primary Key
    "lt" :  [
            {
                  "timestamp"   : epoch1,
                  "coordinates" : "coordinate1"
            },
            {
                  "timestamp"   : epoch2,
                  "coordinates" : "coordinate2"
            },
            ...
        ]
}

The "lt" is location-tracking list, which is intended to store the coordinate values for a userID at different times.

Q1 The requirement is:

  • Store a maximum of 1 days worth of location tracking data per user, with the auto deletion happening only when the new LT coordinate data entry is received
  • What this means is at a time, there could be stale LT data, all of which is of 24 hour duration. However, as soon as a new LT coordinate data comes, deletion of stale entries should happen such that entries older than 24 hours are removed.

I'm clear on how to append entries to a list, or even remove entries at a particular index from a list in dynamoDB.

UpdateExpression : "REMOVE lt[0]" - Remove one element

UpdateExpression : "REMOVE lt[0] lt[1]" - Remove elements 0 and 1

However, now the requirement is to remove entries from the beginning of the list, such that the entries older than 24 hours are removed from it. I've banged my head over this for very long, and there does not seem to be any conditional expression which helps us do that. Am I missing something?

Q2 As a workaround, I changed requirements to:

  • Store last 100 entries into this "lt" list.
  • This is going to keep potentially stale LT data for users in case their LT data is not received

If I receive N new LT points for a user, I want to remove the first "n" entries, if the total entries have become 100 + "n". If total entries are less than or equal to 100, no need to delete the entries.

I can obviously append new N entries into the User item's "lt" list, get that User Item back, find out total number of entries, and then remove the first "n" entries, but that would be inefficient, since I'll have to make two queries, one where I'll have to return the entire "lt" list. It would help if the size of the "lt" list could be retrieved be via some sort of Count construct, is there one.

I want to understand how it should actually be done?

1
  • did you ever manage to remove entries from a list with a condition ?
    – BlueMagma
    Commented Jul 26, 2021 at 14:35

1 Answer 1

-1

You do not need to model locations as a list. You could model them as a map, with HH:MM as the key of the map. In your update expression, just SET lt.#hhmm = :coord with ExpressionAttributeNames={#hhmm:"16:05"} and ExpressionAttributeValues={:coord:"0,0"}. If you record location once per minute, that means 24 * 60 = 1440 entries in the lt map. If each coordinate pair is 19 characters long, you have approximately 30 bytes per entry or around 43 KB per person if you record once per minute.

Using the scheme above, it would cost around 43 WCU per minute, or less than 1 WCU per second per user to maintain current location with minute granularity. This is kind of high for one customer. Instead, you could split up the user location item into 30 minute buckets, using 48 such items to cover a 24 hour span per person. Thus, the UpdateItem write cost would be 1 WCU per minute or 60 WCU per hour. The hash key would be of a form like <user_id>_HH:<00 or 30>.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.