Is there a 'cookbook' way of resampling a DataFrame with (semi)irregular periods?
I have a dataset at a daily interval and want it to resample to what sometimes (in scientific literature) is named dekad's. I dont think there is a proper English term for it but its basically chopping a month in three ~ten-day parts where the third is a remainder of anything between 8 and 11 days.
I came up with two solutions myself, a specific one for this case and a more general one for any irregular periods. But both arent really good, so im curiuous how others handle these type of situations.
Lets start with creating some sample data:
import pandas as pd
begin = pd.datetime(2013,1,1)
end = pd.datetime(2013,2,20)
dtrange = pd.date_range(begin, end)
p1 = np.random.rand(len(dtrange)) + 5
p2 = np.random.rand(len(dtrange)) + 10
df = pd.DataFrame({'p1': p1, 'p2': p2}, index=dtrange)
The first thing i came up with is grouping by individual months (YYYYMM) and then slicing it manually. Like:
def to_dec1(data, func):
# create the indexes, start of the ~10day period
idx1 = pd.datetime(data.index[0].year, data.index[0].month, 1)
idx2 = idx1 + datetime.timedelta(days=10)
idx3 = idx2 + datetime.timedelta(days=10)
# slice the period and perform function
oneday = datetime.timedelta(days=1)
fir = func(data.ix[:idx2 - oneday].values, axis=0)
sec = func(data.ix[idx2:idx3 - oneday].values, axis=0)
thi = func(data.ix[idx3:].values, axis=0)
return pd.DataFrame([fir,sec,thi], index=[idx1,idx2,idx3], columns=data.columns)
dfmean = df.groupby(lambda x: x.strftime('%Y%m'), group_keys=False).apply(to_dec1, np.mean)
Which results in:
print dfmean
p1 p2
2013-01-01 5.436778 10.409845
2013-01-11 5.534509 10.482231
2013-01-21 5.449058 10.454777
2013-02-01 5.685700 10.422697
2013-02-11 5.578137 10.532180
2013-02-21 NaN NaN
Note that you always get a full month of 'dekads' in return, its not a problem and easy to remove if needed.
The other solution works by providing a range of dates at which you chop up the DataFrame and perform a function on each segment. Its more flexible in terms of the periods you want.
def to_dec2(data, dts, func):
chucks = []
for n,start in enumerate(dts[:-1]):
end = dts[n+1] - datetime.timedelta(days=1)
chucks.append(func(data.ix[start:end].values, axis=0))
return pd.DataFrame(chucks, index=dts[:-1], columns=data.columns)
dfmean2 = to_dec2(df, dfmean.index, np.mean)
Note that im using the index of the previous result as the range of dates to save some time 'building' it myself.
What would be the best way of handling these cases? Is there perhaps a bit more build-in method in Pandas?