6

I have two columns in my dataframe - Start date and Finish date (It is a timestamp ). I would like to calculate runtime by finding the difference. When I run the code below, I get the error: TypeError: cannot subtract DatetimeArray from ndarray. I am also unable to split the finished date column into date and time so I can calculate runtime in another manner.

# calculate runtime 
pd.to_datetime(df.start_date)
pd.to_datetime(df.finish_date)

diff = pd.to_datetime(df.finish_date) - pd.to_datetime(df.start_date)

This is how the dataframe looks like: df

file_name                unique_id                                      start_date                         finish_date
        
file1         1716b94a8d8d83c8fffe4bdd14d536ae1ee9cba6bf17e6...   2020-09-03T16:18:38.929863799Z          2020-09-03T16:20:17.615093582Z
        
file2         87ff84ab119b798312230fceb3a8730fe74669a373650a...   2020-09-03T16:26:25.075167073Z          2020-09-04T00:04:39.702686798Z

How can I -

  1. either find the difference between finish & start date?
  2. split the columns into date & time.

1 Answer 1

6

Assume you want to calculate the number of days between the dates, then this is one solution:

import datetime as dt
diff = (pd.to_datetime(df.finish_date) - pd.to_datetime(df.start_date)).dt.days

EDIT

Another alternative is

Start = pd.to_datetime(df.finish_date)
End  = pd.to_datetime(df.start_date)
End.subtract(Start)

Example: Here I choose to compute the difference between dates in a df and now.

metric_id device_id            timestamp  cpu_5min  vol_max
0          device_1  2020-12-04 05:15:00     116.0    734.0
1          device_1  2020-12-04 05:30:00     213.0    325.0
2          device_1  2020-12-04 05:35:00     427.0    668.0
3          device_2  2020-12-04 05:15:00     540.0      NaN
4          device_2  2020-12-04 05:30:00     127.0      NaN
5          device_2  2020-12-04 05:35:00     654.0      NaN

and

df['tDATE'] = pd.to_datetime(df['timestamp'])
df['DIFF1'] = (df['tDATE'] - dt.datetime.now()).dt.days  #method 11
df['DIFF2'] = df['tDATE'].subtract(dt.datetime.now()) #method2

which returns

metric_id device_id            timestamp  cpu_5min  vol_max  \
0          device_1  2020-12-04 05:15:00     116.0    734.0   
1          device_1  2020-12-04 05:30:00     213.0    325.0   
2          device_1  2020-12-04 05:35:00     427.0    668.0   
3          device_2  2020-12-04 05:15:00     540.0      NaN   
4          device_2  2020-12-04 05:30:00     127.0      NaN   
5          device_2  2020-12-04 05:35:00     654.0      NaN   

metric_id               tDATE  difd  DIFF1                     DIFF2  
0         2020-12-04 05:15:00   -14    -14 -14 days +22:13:26.627607  
1         2020-12-04 05:30:00   -14    -14 -14 days +22:28:26.627607  
2         2020-12-04 05:35:00   -14    -14 -14 days +22:33:26.627607  
3         2020-12-04 05:15:00   -14    -14 -14 days +22:13:26.627607  
4         2020-12-04 05:30:00   -14    -14 -14 days +22:28:26.627607  
5         2020-12-04 05:35:00   -14    -14 -14 days +22:33:26.627607  

EDIT: Working with Timestamps

From your comments below it is now evident that the examples above need preparation since you are working with this Timepstamps. As a note, this is why giving enough information when asking the question is important (e.g. what kind of data you are dealing with). This is even moe important when it comes to dates as there are many formats. Here is an example with the date format you gave in the comments:

import datetime as dt

Date = '2020-09-03T16:18:38.929863799Z'
Date2  = '2020-10-03T16:18:38.929863799Z'

What you have here are Timestamps, so your first step is to convert them to datetime and then use to_pydate (It use to be called Timestamp.to_datetime() but is now deprecated.

Date = pd.to_datetime(Date)
Date2  = pd.to_datetime(Date2)

DATE_1 = Date.to_pydatetime()
DATE_2 = Date2.to_pydatetime()

after which you can compute the difference

DIFF = (pd.to_datetime(DATE_1) -pd.to_datetime(DATE_2))

which is Timedelta('-30 days +00:00:00')

6
  • More like the minutes it took, not days.
    – py_noob
    Commented Dec 17, 2020 at 5:32
  • 3
    Tried it out, I get the same error : cannot subtract DatetimeArray from ndarray
    – py_noob
    Commented Dec 17, 2020 at 5:33
  • 2
    Thanks for the detailed explanation. I appreciate it. I get an error: UFuncTypeError: ufunc 'subtract' cannot use operands with types dtype('<M8[ns]') and dtype('O') Something about the datatype of the datetime timestamp in my dataframe is not right seems to me. I have never seen so many different errors arising from this.
    – py_noob
    Commented Dec 17, 2020 at 16:03
  • How do you work your example with the time in the format 2020-09-03T16:18:38.929863799Z or get the timestamp to the format you want to be able to use the subtract function?
    – py_noob
    Commented Dec 17, 2020 at 16:05
  • I have completed my answer with dealing with timestamps. If it is what you were looking for, check it off so it disappears from the unanswered questions. Commented Dec 18, 2020 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.