8

I am trying to use Mathematica to analyse some raw data. I'd like to be able to dynamically display the range of data I'm interested in using Manipulate and ListLinePlot, but the plot rendering is extremely slow. How can I speed it up?

Here are some additional details. An external text file stores the raw data: the first column is a timestamp, the second, third and fourth columns are data readings, for example:

1309555993069, -2.369941, 6.129157, 6.823794
1309555993122, -2.260978, 6.170018, 7.014479
1309555993183, -2.070293, 6.129157, 6.823794
1309555993242, -1.988571, 6.238119, 7.123442

A single data file contains up to 2·106 lines. To display, for example, the second column, I use:

x = Import["path/to/datafile"];
ListLinePlot[x[[All, {1, 2}]]]

The execution time of this operation is unbearably long. To display a variable range of data I tried to use Manipulate:

Manipulate[ListLinePlot[Take[x, numrows][[All, {1, 2}]]], {numrows, 1, Length[x]}]

This instruction works, but it quickly crawls when I try to display more than few thousand lines. How can I speed it up?

Some additional details:

  • MATLAB displays the same amount of data on the same computer almost instantaneously, thus the raw data size shouldn't be an issue.
  • I already tried to turn off graphics antialiasing, but it didn't impact rendering speed at all.
  • Using DataRange to avoid Take doesn't help.
  • Using MaxPlotPoints distorts too much the plot to be useful.
  • Not using Take in Manipulate doesn't help.
  • The rendering seems to take huge amount of time. Running Timing[ListLinePlot[Take[x,100000][[All, {1, 2}]]]] returns 0.33: this means that the evaluation of Take by itself is almost instantaneous, is the plot rendering that slows everything down.
  • I am running Mathematica on Ubuntu Linux 11.10 using the fglrx drivers. Forcing Mathematica to use mesa drivers didn't help.

Any hint?

4 Answers 4

12

If your goal is to just visualize your data quickly but properly, you can use the following trick, which I am constantly using.

I partition the data into a number of blocks corresponding roughly to the resolution of my screen (usually 1000 or less), more detail cannot be displayed anyway. Then I determine the Min and Max of each block, and draw a zig-zag line from min to max to min to max... The result will look exactly like the original data. You can however not "zoom in", as you would then see the zig-zag line (e.g. when exporting to high-res pdf). Then you need to use a larger number of blocks.

rv = RandomVariate[ExponentialDistribution[2], 100000];

ListLinePlot[rv, PlotRange -> All] (* original, slow *)
ListLinePlot[rv, PlotRange -> All, MaxPlotPoints -> 1000] (* fast but distorted *)

numberOfBlocks = 1000;

ListLinePlot[Riffle @@ Through[{Min /@ # &, Max /@ # &}[
   Partition[rv,Floor[Length[rv]/numberOfBlocks]]]], PlotRange -> All]

You can add the DataRange->{...} option to label the x-axis appropriately.

Hope this helps!

EDIT: See also this similar question on Mathematica Stackexchange: https://mathematica.stackexchange.com/q/140/58

1
  • Nice --- I didn't see this when I posted my Jul 26 answer.
    – Szabolcs
    Jul 27, 2011 at 15:28
7

I haven't tested extensively this on my machine (I have a Mac, so I can't rule out Linux-specific issues). but a couple of points occur to me. The following was pretty quick for me, but obviously slower than if the data set was smaller. You are plotting hundreds of thousands of data points.

data = Accumulate@RandomVariate[NormalDistribution[], 200000];
Manipulate[ListLinePlot[Take[data, n]], {n, 1, Length[data]}]
  1. In a Manipulate, you are allowing the amount of data shown with Take to vary arbitrarily. Try only incrementing numrows every 100 or so points, so there is less to render.
  2. Try using the ContinuousAction->False option (see documentation) (I see @Szabolcs had the same idea as I was typing.
  3. I was about to suggest MaxPlotPoints, but instead try the PerformanceGoal ->"Speed" option. (see documentation)
9
  • Using a reasonable value for MaxPlotPoints and using PerformanceGoal did the trick. The rendering is still worse than the MATLAB's one, but it is much better than before, thanks! Jul 3, 2011 at 15:30
  • What is PerformanceGoal -> "Speed" actually doing? I thought it just reduced MaxPlotPoints.
    – Szabolcs
    Jul 3, 2011 at 15:41
  • @giuseppe what value of MaxPlotPoints did you use? I feel the number of pixels in your figure (ImageSize) or on your screen would be a good choice. At 1000, Verbeia's data plots extremely fast. It's on my screen before my finger is able to get off the enter key. Quality is very good. Jul 4, 2011 at 0:09
  • @Sjoerd that's what I said in my other comment, but actually if the fluctuations in the data are very strong, then any reduction in MaxPlotPoints can have an effect on appearance, regardless of pixel width. On the other hand with such strong fluctuations one can't see a lot on a small figure.
    – Szabolcs
    Jul 4, 2011 at 7:33
  • 1
    @giuseppe, you can always revert to the pre-version-6 graphics behaviour with << Version5`Graphics` , and use ListPlot[data, PlotJoined->True], which will give you very fast rendering for a million datapoints. But this has so many other disadvantages and incompatibilities with post-6 features that it's very difficult to make it useful.
    – Szabolcs
    Jul 4, 2011 at 8:23
5

I also noticed that occasionally Mathematica will take too long to render graphics. Actually it must be some translation step from a Mathematica Graphics expression to some other representation that takes long because once rendered, resizing (and thus re-rendering) the graphic is much faster. Pre-version-6 graphics rendering used to be faster for many examples (but also lacks a lot of functionality that 6+ have).

Some ideas about what you could do:

  1. Use the MaxPlotPoints option of ListLinePlot to reduce the data before plotting. It might not make a difference in looks if its downsampled. The Method option should choose the downsample algorithm, but I can't find any docs for it (anyone?)

  2. Use ContinuousAction -> False in Manipulate to stop it from recomputing everything in real time as you drag the sliders.

5
  • +1 just for mentioning MaxPlotPoints, which I did not know about
    – acl
    Jul 2, 2011 at 23:33
  • And +1 for beating me to ContinuousAction :)
    – Verbeia
    Jul 2, 2011 at 23:34
  • I am sorry that my question wasn't clear. ContinuousAction -> False would certainly help, but the plotting procedure is extremely slow even for ListLinePlot[Take[x, 100000][[All, {1, 2}]]], and MaxPlotPoints distorts the plot too much to be useful (strangely, even if I use it the plot time doesn't decrease appreciably) Jul 2, 2011 at 23:48
  • 1
    Are you sure you can't use MaxPlotPoints? The graphic is only so many pixels wide (certainly much much less than 10^6!), and it makes no sense to use more points than the pixel-width at most.
    – Szabolcs
    Jul 3, 2011 at 5:24
  • You're right, I was using a value too high for MaxPlotPoints. The rendering is of lower quality compared to MATLAB's one and is much slower, anyway it is better than before. I am accepting Verbeia's answer because he suggested using PerformanceGoal, that seems to help. Thanks for your answer! Jul 3, 2011 at 15:35
5

Another idea here is using the Ramer–Douglas–Peucker algorithm to reduce the number of data points before plotting. This will likely preserve the shape of the data better. I don't know if you still need this so I won't provide an implementation.

1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.