34

I am trying to create a histogram on a continuous value column Trip_distance in a large 1.4M row pandas dataframe. Wrote the following code:

fig = plt.figure(figsize=(17,10))
trip_data.hist(column="Trip_distance")
plt.xlabel("Trip_distance",fontsize=15)
plt.ylabel("Frequency",fontsize=15)
plt.xlim([0.0,100.0])
#plt.legend(loc='center left', bbox_to_anchor=(1.0, 0.5))

But I am not sure why all values give the same frequency plot which shouldn't be the case. What's wrong with the code?

Test data:

    VendorID    lpep_pickup_datetime    Lpep_dropoff_datetime   Store_and_fwd_flag  RateCodeID  Pickup_longitude    Pickup_latitude Dropoff_longitude   Dropoff_latitude    Passenger_count Trip_distance   Fare_amount Extra   MTA_tax Tip_amount  Tolls_amount    Ehail_fee   improvement_surcharge   Total_amount    Payment_type    Trip_type
0   2   2015-09-01 00:02:34 2015-09-01 00:02:38 N   5   -73.979485  40.684956   -73.979431  40.685020   1   0.00    7.8 0.0 0.0 1.95    0.0 NaN 0.0 9.75    1   2.0
1   2   2015-09-01 00:04:20 2015-09-01 00:04:24 N   5   -74.010796  40.912216   -74.010780  40.912212   1   0.00    45.0    0.0 0.0 0.00    0.0 NaN 0.0 45.00   1   2.0
2   2   2015-09-01 00:01:50 2015-09-01 00:04:24 N   1   -73.921410  40.766708   -73.914413  40.764687   1   0.59    4.0 0.5 0.5 0.50    0.0 NaN 0.3 5.80    1   1.0
3   2   2015-09-01 00:02:36 2015-09-01 00:06:42 N   1   -73.921387  40.766678   -73.931427  40.771584   1   0.74    5.0 0.5 0.5 0.00    0.0 NaN 0.3 6.30    2   1.0
4   2   2015-09-01 00:00:14 2015-09-01 00:04:20 N   1   -73.955482  40.714046   -73.944412  40.714729   1   0.61    5.0 0.5 0.5 0.00    0.0 NaN 0.3 6.30    2   1.0
5   2   2015-09-01 00:00:39 2015-09-01 00:05:20 N   1   -73.945297  40.808186   -73.937668  40.821198   1   1.07    5.5 0.5 0.5 1.36    0.0 NaN 0.3 8.16    1   1.0
6   2   2015-09-01 00:00:52 2015-09-01 00:05:50 N   1   -73.890877  40.746426   -73.876923  40.756306   1   1.43    6.5 0.5 0.5 0.00    0.0 NaN 0.3 7.80    1   1.0
7   2   2015-09-01 00:02:15 2015-09-01 00:05:34 N   1   -73.946701  40.797321   -73.937645  40.804516   1   0.90    5.0 0.5 0.5 0.00    0.0 NaN 0.3 6.30    2   1.0
8   2   2015-09-01 00:02:36 2015-09-01 00:07:20 N   1   -73.963150  40.693829   -73.956787  40.680531   1   1.33    6.0 0.5 0.5 1.46    0.0 NaN 0.3 8.76    1   1.0
9   2   2015-09-01 00:02:13 2015-09-01 00:07:23 N   1   -73.896820  40.746128   -73.888626  40.752724   1   0.84    5.5 0.5 0.5 0.00    0.0 NaN 0.3 6.80    2   1.0
In [ ]:

Trip_distance column 

0     0.00
1     0.00
2     0.59
3     0.74
4     0.61
5     1.07
6     1.43
7     0.90
8     1.33
9     0.84
10    0.80
11    0.70
12    1.01
13    0.39
14    0.56
Name: Trip_distance, dtype: float64

enter image description here

After 100 bins:

enter image description here

5
  • Can you include a test data set that is giving the unexpected result, and describe the result that you would like to see?
    – johnchase
    Feb 27, 2017 at 22:04
  • Added. If you see my values of Trip_distance column has various values and still the graph I get has same frequency count?
    – Baktaawar
    Feb 27, 2017 at 22:11
  • 1
    Which value range are the trip distances? The datetime columns are hard to copy. Feb 27, 2017 at 22:22
  • 1
    And just how much data is there for you to get a frequency of 140000? I suspect there might simply be too much data. Feb 27, 2017 at 22:27
  • the total rows are 1.4MM. I updated the trip_distance sample values
    – Baktaawar
    Feb 27, 2017 at 22:28

2 Answers 2

42

EDIT:

After your comments this actually makes perfect sense why you don't get a histogram of each different value. There are 1.4 million rows, and ten discrete buckets. So apparently each bucket is exactly 10% (to within what you can see in the plot).


A quick rerun of your data:

In [25]: df.hist(column='Trip_distance')

enter image description here

Prints out absolutely fine.

The df.hist function comes with an optional keyword argument bins=10 which buckets the data into discrete bins. With only 10 discrete bins and a more or less homogeneous distribution of hundreds of thousands of rows, you might not be able to see the difference in the ten different bins in your low resolution plot:

In [34]: df.hist(column='Trip_distance', bins=50)

enter image description here

4
  • I just updated my question with graph after 100 bins. Doesn't look that great though
    – Baktaawar
    Feb 27, 2017 at 22:45
  • I mean, definitely drop the xlim=[0,100] you don't have trips above 60 Feb 27, 2017 at 22:47
  • I just did till 40.
    – Baktaawar
    Feb 27, 2017 at 22:48
  • I have a question, how could you save this figure?
    – sikisis
    Mar 26, 2020 at 2:29
4

Here's another way to plot the data, involves turning the date_time into an index, this might help you for future slicing

#convert column to datetime
trip_data['lpep_pickup_datetime'] = pd.to_datetime(trip_data['lpep_pickup_datetime'])
#turn the datetime to an index
trip_data.index = trip_data['lpep_pickup_datetime']
#Plot
trip_data['Trip_distance'].plot(kind='hist')
plt.show()
1
  • 1
    you can specify bins (kind='hist', bins=20) Feb 27, 2017 at 23:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.