8

I have a DataFrame with some (more-sensical) data in the following form:

In[67] df
Out[67]: 
                             latency
timestamp                           
2016-09-15 00:00:00.000000  0.042731
2016-09-15 00:16:24.376901  0.930874
2016-09-15 00:33:19.268295  0.425996
2016-09-15 00:51:30.956065  0.570245
2016-09-15 01:09:23.905364  0.044203
                             ...
2017-01-13 13:08:31.707328  0.071137
2017-01-13 13:25:41.154199  0.322872
2017-01-13 13:38:19.732391  0.193918
2017-01-13 13:57:36.687049  0.999191

So it spans about 50 days, and the timestamps are not at the same time every day. I would like to overlay some plots for each day, that is, inspect the time series of each day on the same plot. 50 days may be too many lines, but I think there is a kind of "daily seasonality" which I would like to investigate, and this seems like a useful visualization before anything more rigorous.

How do I overlay this data on the same plot representing a "single-day" time period?


My thoughts

I am not yet very familiar with Pandas, but I managed to group my data into daily bins with

In[67]: df.groupby(pd.TimeGrouper('D'))
Out[68]: <pandas.core.groupby.DataFrameGroupBy object at 0x000000B698CD34E0>

Now I've been trying to determine how I am supposed to create a new DataFrame structure such that the plots can be overlayed by day. This the fundamental thing I can't figure out - how can I utilize a DataFrameGroupBy object to overlay the plots? A very rudimentary-seeming approach would be to just iterate over each GroupBy object, but my issue with doing so has been configuring the x-axis such that it only displays a "daily time period" independent of the particular day, instead of capturing the entire timestamp.

Splitting the data up into separate frames and calling them in the same figure with some kind of date coercion to use the approach in this more general answer doesn't seem very good to me.


You can generate pseudo-data similarly with something like this:

import datetime 

start_date = datetime.datetime(2016, 9, 15)
end_date = datetime.datetime.now()

dts = []
cur_date = start_date
while cur_date < end_date:
    dts.append((cur_date, np.random.rand()))
    cur_date = cur_date + datetime.timedelta(minutes=np.random.uniform(10, 20))

4 Answers 4

9
+50

Consider the dataframe df (generated mostly from OP provided code)

import datetime 

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

start_date = datetime.datetime(2016, 9, 15)
end_date = datetime.datetime.now()

dts = []
cur_date = start_date
while cur_date < end_date:
    dts.append((cur_date, np.random.rand()))
    cur_date = cur_date + datetime.timedelta(minutes=np.random.uniform(10, 20))


df = pd.DataFrame(dts, columns=['Date', 'Value']).set_index('Date')

The real trick is splitting the index into date and time components and unstacking. Then interpolate to fill in missing values

d1 = df.copy()
d1.index = [d1.index.time, d1.index.date]
d1 = d1.Value.unstack().interpolate()

From here we can d1.plot(legend=0)

ax = d1.plot(legend=0)
ax.figure.autofmt_xdate()

enter image description here

But that isn't very helpful.


You might try something like this... hopefully this helps

n, m = len(d1.columns) // 7 // 4 + 1, 4
fig, axes = plt.subplots(n, m, figsize=(10, 15), sharex=False)

for i, (w, g) in enumerate(d1.T.groupby(pd.TimeGrouper('W'))):
    r, c = i // m, i % m
    ax = g.T.plot(ax=axes[r, c], title=w, legend=0)

fig.autofmt_xdate()

enter image description here


How to do it over weeks

  • create a multi index
    • include the period representing the week
    • include the day of the week
    • include the time of day
  • unstack to get weekly periods into columns
  • still not convinced of the axis format

d2 = df.copy()

idx = df.index
d2.index = [idx.weekday_name, idx.time, idx.to_period('W').rename('Week')]

ax = d2.Value.unstack().interpolate().iloc[:, :2].plot()
ax.figure.autofmt_xdate()

enter image description here

6
  • 1
    Apologies for not getting back to you sooner @piRSquared, but this answer with a few tweaks was incredibly helpful, thank you. Jan 17, 2017 at 19:10
  • @EricHansen No Problem. Glad I could help
    – piRSquared
    Jan 17, 2017 at 19:11
  • I placed a bounty to reward you with. I know it is not what I originally asked and I don't expect an answer, but out of curiosity, if I wanted to superimpose on each week instead of each day, would it be as simple as tweaking the indexing? I figured I would just have to index on the time and then the week, however this doesn't work as then I have duplicate entries, which makes sense. I've been trying to wrap my head around the state model for reindexing with a DatetimeIndex and have been struggling. Jan 18, 2017 at 23:16
  • @EricHansen that is a really nice gesture and thank you. It wasn't necessary but I appreciate it none the less. I've updated my post, with an idea how to do the weekly thing. It's half baked, but it's a start. I'd suggest asking this as a question and maybe even tweaking my suggestion to suit your needs so you can share with others what you've discovered as your own answer to your question.
    – piRSquared
    Jan 19, 2017 at 0:36
  • That is great, thanks. And sure, I will do so tomorrow when I have a bit of time. I'll just say that while learning pandas over the past few weeks I have learned from several of your answers around the site so I really appreciate it. Jan 19, 2017 at 0:40
0

You have not mentioned what opeation do you intend on the latencies grouped by day. Say if you take mean values, you can plot a simple line graph like this:

df = pd.DataFrame(dts)
df.columns = ['Timestamp', 'Latency']

df.groupby(pd.TimeGrouper(key='Timestamp',freq='D')).mean().plot()
1
  • Sorry if I was unclear - I don't want any aggregation done. I want a plot with an x-axis from time 0 to time 24h, and I want the time series for each day on that same plot. I understand how to aggregate and then plot. Jan 13, 2017 at 21:06
0

If you add separate column columns for date and time then you just have to plot time against latency for each date.

df = df.assign(date=df.index.date, time=df.index.time)
for date in df.date.unique():
    plt.plot('time', 'latency', data=df[df.date == date])
    plt.xlabel('latency')
0

I recently had to do a very similar plot using random timestamped events for data I was analyzing.

You need to add another column in your dataframe for finding elapsed time

Please make sure your timestamp data is a python datetime object first, then do

df['Elapsed_Time'] = df['timestamp'] - df['timestamp'][0]
df['Elapsed_Time'] = df['Elapsed_Time'] / datetime.timedelta(days=1)

Now you should have a dataframe with elapsed time column(something like the following. I am using my own dataset to show you what i mean)

enter image description here

Also, if you want a plot every hour instead of every day. Then just use hours instead of days in the line

df['Elapsed_Time'] = df['Elapsed_Time'] / datetime.timedelta(hours=1)

next steps: plotting

The idea is to go through the dataset row by row and aggregate data that fall in a day timeframe and then append to a list

latency = []
next_day = 1
inds = []
for (i, t) in enumerate(list(df['Elapsed_Time'])):
    if t < next_day:
        inds.append(i)
    else:
        latency.append(df.iloc[inds]))
        next_day += 1
        inds = []
plt.plot(latency, "bo--", label="latencyperday")

This is the end result(using my own dataset to show you what it would look like). Hope this helps

enter image description here

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.