3

Suppose I have the list data:

import numpy as np
import datetime

np.random.seed(0)
aux = [10,30,50,60,70,110,120]
base = datetime.datetime(2018, 1, 1, 22, 34, 20)
data = [[base + datetime.timedelta(seconds=s), 
         round(np.random.rand(),3)] for s in aux]

This returns:

data == 

[[datetime.datetime(2018, 1, 1, 22, 34, 30), 0.549],
 [datetime.datetime(2018, 1, 1, 22, 34, 50), 0.715],
 [datetime.datetime(2018, 1, 1, 22, 35, 10), 0.603],
 [datetime.datetime(2018, 1, 1, 22, 35, 20), 0.545],
 [datetime.datetime(2018, 1, 1, 22, 35, 30), 0.424],
 [datetime.datetime(2018, 1, 1, 22, 36, 10), 0.646],
 [datetime.datetime(2018, 1, 1, 22, 36, 20), 0.438]]

What I want to do is fill the spaces where the gaps in the dates are greater than10 seconds using the last previous value. For this example, the output should be:

desired_output ==

[[datetime.datetime(2018, 1, 1, 22, 34, 30), 0.549],
 [datetime.datetime(2018, 1, 1, 22, 34, 40), 0.549],
 [datetime.datetime(2018, 1, 1, 22, 34, 50), 0.715],
 [datetime.datetime(2018, 1, 1, 22, 35), 0.715],
 [datetime.datetime(2018, 1, 1, 22, 35, 10), 0.603],
 [datetime.datetime(2018, 1, 1, 22, 35, 20), 0.545],
 [datetime.datetime(2018, 1, 1, 22, 35, 30), 0.424],
 [datetime.datetime(2018, 1, 1, 22, 35, 40), 0.424],
 [datetime.datetime(2018, 1, 1, 22, 35, 50), 0.424],
 [datetime.datetime(2018, 1, 1, 22, 36), 0.424],
 [datetime.datetime(2018, 1, 1, 22, 36, 10), 0.646],
 [datetime.datetime(2018, 1, 1, 22, 36, 20), 0.438]]

I can't think of any smart way to do this. All dates are separated by multiples of 10 seconds. Any ideas?

1 Answer 1

2

Option 1: with Pandas

If you're open to using Pandas, it makes reindexing operations like this easy:

>>> import pandas as pd
>>> df = pd.DataFrame(data, columns=['date', 'value'])
>>> ridx = df.set_index('date').asfreq('10s').ffill().reset_index()
>>> ridx
                  date  value
0  2018-01-01 22:34:30  0.549
1  2018-01-01 22:34:40  0.549
2  2018-01-01 22:34:50  0.715
3  2018-01-01 22:35:00  0.715
4  2018-01-01 22:35:10  0.603
5  2018-01-01 22:35:20  0.545
6  2018-01-01 22:35:30  0.424
7  2018-01-01 22:35:40  0.424
8  2018-01-01 22:35:50  0.424
9  2018-01-01 22:36:00  0.424
10 2018-01-01 22:36:10  0.646
11 2018-01-01 22:36:20  0.438

.asfreq('10s') will fill the missing 10-second intervals. .ffill() means "forward-fill" missing values with the last-seen valid value.

To get back to the data structure that you have now (though note that the elements will be 2-tuples, rather then lists of length 2):

>>> native_ridx = list(zip(ridx['date'].dt.to_pydatetime().tolist(), ridx['value']))
>>> from pprint import pprint
>>> pprint(native_ridx[:5])
[(datetime.datetime(2018, 1, 1, 22, 34, 30), 0.549),
 (datetime.datetime(2018, 1, 1, 22, 34, 40), 0.549),
 (datetime.datetime(2018, 1, 1, 22, 34, 50), 0.715),
 (datetime.datetime(2018, 1, 1, 22, 35), 0.715),
 (datetime.datetime(2018, 1, 1, 22, 35, 10), 0.603)]

To confirm:

>>> assert all(tuple(i) == j for i, j in zip(desired_output, native_ridx))

Option 2: Native Python

import datetime

def make_daterange(
    start: datetime.datetime,
    end: datetime.datetime,
    incr=datetime.timedelta(seconds=10)
):
    yield start
    while start < end:
        start += incr
        yield start

def reindex_ffill(data: list, incr=datetime.timedelta(seconds=10)):
    dates, _ = zip(*data)
    data = dict(data)
    start, end = min(dates), max(dates)
    daterng = make_daterange(start, end, incr)
    # If initial value is not valid, the element at [0][0] will be NaN
    lastvalid = np.nan
    get = data.get
    for date in daterng:
        value = get(date)
        if value:
            yield date, value
            lastvalid = value
        else:
            yield date, lastvalid

Example:

>>> pynative_ridx = list(reindex_ffill(data))
>>> assert all(tuple(i) == j for i, j in zip(desired_output, pynative_ridx))
2
  • I've tried the first solution and it works perfectly with the example I provided. However, with my real dataset, df.set_index('date').asfreq('10s') returns an empty dataframe. Do you know why this could happen?
    – Tendero
    Commented Oct 9, 2018 at 20:13
  • 1
    I got it! The dates weren't sorted and asfreq was not working because of that. Replacing that line by df.set_index('date').sort_index().asfreq('10s').ffill().reset_index() worked alright. Thanks for your solution, it's very fast and readable :)
    – Tendero
    Commented Oct 9, 2018 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.