5

Problem Constraints

  • Size of the data set, but not the data itself, is known.
  • Data set grows by one data point at a time.
  • Trend line is graphed one data point at a time (using a spline/Bezier curve).

Graphs

The collage below shows data sets with reasonably accurate trend lines:

enter image description here

The graphs are:

  • Upper-left. By hour, with ~24 data points.
  • Upper-right. By day for one year, with ~365 data points.
  • Lower-left. By week for one year, with ~52 data points.
  • Lower-right. By month for one year, with ~12 data points.

User Inputs

The user can select:

  • the type of time series (hourly, daily, monthly, quarterly, annual); and
  • the start and end dates for the time series.

For example, the user could select a daily report for 30 days in June.

Trend Weight

To calculate the window size (i.e., the number of data points to average when calculating the trend line), the following expression is used:

data points / trend weight

Where data points is derived from user inputs and trend weight is 6.4. Even though a trend weight of 6.4 produces good fits, it is rather arbitrary, and might not be appropriate for different user inputs.

Question

How should trend weight be calculated given the constraints of this problem?

2
  • 1
    Are you fitting a trend line, or just calculating the moving average within a window and then putting a spline through the values?
    – Aniko
    Mar 24, 2010 at 12:35
  • 1
    I believe I am after a smoothing spline. en.wikipedia.org/wiki/Smoothing_spline Mar 24, 2010 at 15:54

3 Answers 3

1

Based on the looks of the graphs I would say you have too many points for your 12 point graph (it is just a spline of the points given... which is visually pleasing, but actually does more harm than good when trying to understand the trend) and too few points for your 365 point graph. Perhaps try doing something a little exponential like:

(Data points)^1.2/14.1

I do realize this is even more arbitrary than what you already have, but arbitrary isn't the worst thing in the world.

(I got 14.1 by trying to keep the 52 point graph fixed, since that one looks nice, by taking (52^(1.2)/52)*6.4=14.1. You using this technique you could try other powers besides 1.2 to see what you visually get.

Dan

1
  • I may have phrased my suggestion wrong. I'm proposing alternative window size calculations.
    – Dan
    Mar 24, 2010 at 18:12
0

I voted this up for the quality of your results and the clarity of your write-up. I wish I could offer an answer that could improve on your already excellent work.

I fear that it might be a matter of trial and error with the trend weight until you see an improved fit.

It could be that you could make this an input from users as well: allow them to fiddle with the value, given realistic constraints, until they get satisfactory values.

I also wondered if the weight would be different for each graph, since the number of points in each is different. Are you trying to get a single weighting that works for all graphs?

Excellent work; a nice question. Well done. I wish I was more helpful. Perhaps someone else will have more wisdom to impart than I do.

2
  • Turns out that the "real way" to solve this problem would take a book to explain. Essentially, though, it involves calling R functions in the database. The R functions then perform the statistical analysis, and provide an extra column of data back to the report. Calculating the trend line in iReport is not a good idea. Dec 27, 2010 at 0:26
  • Excellent, I'll be sure to give it a look.
    – duffymo
    Mar 4, 2013 at 17:11
0

It might look like the trend lines are accurate in those 4 graphs but its really quite off. (This is best seen in the begging of the lower left one and the beginning of the upper right. I would think that you would want to use no less than half of your points when finding the trend line (though really you should use much more than half). I would suggest a Trend Weight of 2 at a maximum. Though really you ought to stick closer to the 1-1.5 range. Since it is arbitrary i would suggest you give your user an "accuracy of trend line" slider that they can use where the most accurate setting uses a trend weight of 1 and the least accurate uses a weight of #of data points +1. This would use 0 points (amusing you always round down) and, i would assume, though your statistics software might be different, will generate a strait horizontal line.

3
  • Hi, David. Thanks for the help. Due to the API, each data point must become some point on the trend line. Using a trend weight of 2 won't work. The reason the upper-right is off at the beginning is because there are few data points between January and March, which is not the case with production data. I thought about letting them pick a value for the trend line's weight (with a suggested value), but was hoping there was some formula I could apply. Mar 24, 2010 at 3:22
  • In the one in the upper right it doesn't look like its off for lack of data. Its going way too high without data to get it there. in that first month its peaking above the max for the next month as well as well above the mean for the next month. I would think that the curve should be bellow the blue line in the first month since theres no data in that month to pull it up above the blue line but there is data in the second month to keep it down.
    – David
    Mar 24, 2010 at 13:57
  • I'm shocked that i used to know anything at all about statistics. I don't remember knowing any of this.
    – David
    Apr 5, 2012 at 3:33

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.