1

I am still working on my New York Subway data. I cleaned and wrangled the data in such a fashion that I now have 'Average Entries' and 'Average Exits' per Station per hour (ranging from 0 to 23) separated for weekend and weekday (category variable with two possible values: weekend/weekday).

What I was trying to do is to create a plot with each station being a row, each row having two columns (first for weekday, second for weekend). I would like to plot 'Average Entries' and 'Average Exits' per hour to gain some information about the stations. There are two things of interest here; firstly the sheer numbers to indicate how busy a station is; secondly the ratio between entries and exits for a given hour to indicate if the station is a living area (loads of entries in the morning, loads of exits in the evening) or more of a working area (loads of exits in the morning, entries peeking around 4, 6 and 8 pm or so). Only problem, there are roughly 550 stations.

I tried plotting it with seaborn facetgrid, which cant handle more than a few stations (10 or so) without running into memory issues.

So I was wondering if anybody had a good idea to accomplish what I am trying to do.

Please find attached a notebook (second to last cell shows my attempt of visualizing the data, i.e. the plotting for 4 stations). That clearly wouldn't work for 500+ stations, so maybe 5 stations in a row after all?

The very last cell contains the data for Station R001 as requested in a comment..

https://github.com/FBosler/Udacity/blob/master/Example.ipynb

Any input much appreciated! Fabian

2

3 Answers 3

2

rather than making 550+ subplots see if you can make two big numpy arrays and then use 2 imview subplots, one for weekdays and one for weekends

for the y-values, first find the min (0) and max (10,000?) for your average values, scale these to fit each fake row of, for example, 10px then offset each row in your data by 10px * the row number.

since you want line plots for each of your 24 data points, you'll have to do linear interpolation between your data points in increments of, again for example, 10px so that the final numpy arrays will be 240 x 5500 x 2.

1

A possible way you could do it is to use the ratio of entries to exits per station. Each day/hour could form a column on an image and each row would be a station. As en example:

from matplotlib import pyplot as plt
import random
import numpy as np


all_stations = []

for i in range(550):
    entries = [float(random.randint(0, 50)) for i in range(7*24)] # Data point for each hour over a week
    exits = [float(random.randint(0, 50)) for i in range(7*24)]

    weekend_entries = entries[:2*7]
    weekend_exits = exits[:2*7]

    day_entries = entries[2*7:]
    day_exits = exits[2*7:]

    weekend_ratio = [np.array(en) / np.array(ex) for en, ex in zip(weekend_entries, weekend_exits)]
    day_ratio = [np.array(en) / np.array(ex) for en, ex in zip(day_entries, day_exits)]

    whole_week = weekend_ratio + day_ratio

    all_stations.append(whole_week)

plt.figure()
plt.imshow(all_stations, aspect='auto', interpolation="nearest")
plt.xlabel("Hours")
plt.ylabel("Station number")
plt.title("Entry/exit ratio per station")
plt.colorbar(label="Entry/exit ratio")
# Add some vertical lines to indicate days
for j in range(1, 7):
    plt.plot([j*24]*2, [0, 550], color="black")
plt.xlim(0, 7*24)
plt.ylim(0, 550)
plt.show()

enter image description here

If you would like to show the actual numbers involved an not the ratio, I would consider splitting the data into two, one image for each of the entries and exit data sets. The intensity of each pixel could then be used to inform on the numbers, not ratio.

2
  • I really like your idea of displaying the data as one station being one row. I would however make some adjustments. 1. Hours are categorial (ranging from 0 to 23) 2. Instead of colors I think it would cool to have a mini bar chart for each hour. 3. subdivide the 24 hour categories into entries and exits each, so that each hour column has two subcolumns where there is a red (respectively green) bar indicating average exits (entries) 4. second chart for weekends. Does that make sense? Aug 19, 2016 at 14:18
  • Yes, you could probably fit all that into an image like array but it gets more complicated when you want different colors representing different things. For point 1., you can just re-label the x-axis to reflect your naming catagories, see plt.xlabels(['cata', 'catb', 'catc' etc]). For point 2., you could add another row or split the columns further, I would probably add another row, so each station had one red-row and one green-row which would be easier to compare. To get this to work would be a bit of a hassle, you would need to contruct your own rgb-a image with each pixel customised
    – kezzos
    Aug 22, 2016 at 8:31
1

You're going to have problems displaying them all on a screen no matter what you do unless you have a whole wall of monitors, however to get around the memory constraint, you could rasterize them and save to image files (I would suggest .png for compressability with images of few distinct colors)

What you want for that is pyplot.savefig()

Here's an answer to another question on how to do that, with some tips and tricks

4
  • If I'm just trying to get a feel for the data, this is definitely what I would do. Save the plot for each figure to a file and then browse them like you would a photo gallery. If you want an impressive visualization for presentation, grab an SVG map of the subway system, and write some javascript to have the plot for each station show up when the user clicks or hovers over the station . Not so hard to do. Aug 19, 2016 at 13:55
  • btw, if you make these plots from the notebook, make sure to disable %matplotlib inline or you're going to have memory issues creating all the plots (this is a known notebook issue: github.com/ipython/ipython/issues/7270). Aug 19, 2016 at 14:03
  • @VictorChubukov thank you for the feedback! Your idea obviously is the route a very skilled programmer would take :) Its probably far above my level, but assuming I wanted to do what you are suggesting, how would I go about it/what would I have to learn first? Aug 19, 2016 at 14:28
  • 1
    @FabianBosler there are a lot of options if you go in that direction. my suggestion would be to start with some basic tutorials on D3.js and working with svg. But others may have completely different approaches. Aug 19, 2016 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.