15

Suppose I sample a selection of database records that return the following numbers:

20.50, 80.30, 70.95, 15.25, 99.97, 85.56, 69.77

Is there an algorithm that can be efficiently implemented in PHP to find the outliers (if there are any) from an array of floats based on how far they deviate from the mean?

11
  • 7
    Outlier based on what?
    – dynamic
    Mar 2, 2013 at 13:21
  • perhaps with math extension: http://www.php.net/manual/en/book.stats.php
    – bitWorking
    Mar 2, 2013 at 13:26
  • 1
    @llnk based on the numbers in the given result. From what I understand, the outliers would be based on the inner and outer quartiles...but then again, I readily admit I'm not great at statistical math.
    – eComEvo
    Mar 2, 2013 at 13:29
  • @EcomEvolution There's many ways of determining outliers - do you mean 'n' standard deviations from the mean? Mar 2, 2013 at 13:30
  • @EcomEvolution: "based on the numbers" is pretty generic way to analyize outliers
    – dynamic
    Mar 2, 2013 at 13:31

3 Answers 3

38

Ok let's assume you have your data points in an array like so:

<?php $dataset = array(20.50, 80.30, 70.95, 15.25, 99.97, 85.56, 69.77); ?>

Then you can use the following function (see comments for what is happening) to remove all numbers that fall outside of the mean +/- the standard deviation times a magnitude you set (defaults to 1):

<?php

function remove_outliers($dataset, $magnitude = 1) {

  $count = count($dataset);
  $mean = array_sum($dataset) / $count; // Calculate the mean
  $deviation = sqrt(array_sum(array_map("sd_square", $dataset, array_fill(0, $count, $mean))) / $count) * $magnitude; // Calculate standard deviation and times by magnitude

  return array_filter($dataset, function($x) use ($mean, $deviation) { return ($x <= $mean + $deviation && $x >= $mean - $deviation); }); // Return filtered array of values that lie within $mean +- $deviation.
}

function sd_square($x, $mean) {
  return pow($x - $mean, 2);
} 

?>

For your example this function returns the following with a magnitude of 1:

Array
(
    [1] => 80.3
    [2] => 70.95
    [5] => 85.56
    [6] => 69.77
)
8
  • Thank you! Works quite well. I'm good with algorithms but strangely terrible at math. Not sure how that happened! :)
    – eComEvo
    Mar 2, 2013 at 15:16
  • @EcomEvolution NP... the $filter part was unnecessary didn't realise I posted it so you can remove it. Also I'm not the best at math so I used wikipedia for this. Mar 2, 2013 at 19:28
  • I need a function to remove outliers from an array and I come across your answer. Erm, why not just set $magnitude = 1 in the argument list instead of setting it to NULL first, and then assigning 1 to it afterwards? :)
    – mavili
    Aug 16, 2013 at 15:24
  • 1
    @mavili I actually didn't realise PHP allowed default argument values and was using PHP's type hinting (php.net/manual/en/language.oop5.typehinting.php) to make it an optional parameter. Now I know this has been updated. Aug 16, 2013 at 15:39
  • 1
    genius. this is exactly what i was looking for
    – Francesco
    May 18, 2016 at 19:10
1

For a normally distributed set of data, removes values more than 3 standard deviations from the mean.

<?php
function remove_outliers($array) {
    if(count($array) == 0) {
      return $array;
    }
    $ret = array();
    $mean = array_sum($array)/count($array);
    $stddev = stats_standard_deviation($array);
    $outlier = 3 * $stddev;
    foreach($array as $a) {
        if(!abs($a - $mean) > $outlier) {
            $ret[] = $a;
        }
    }
    return $ret;
}
1
  • I like this and it works, but the client's I'm developing for don't want to have to install a PECL extension. Unfortunately that meant being unable to use the stats library. Thanks for your input and helping me clarify what I was going for!
    – eComEvo
    Mar 2, 2013 at 15:18
0

Topic: Detecting local, additive outliers in unordered arrays by walking a small window through the array and calculating the standard deviation for a certain range of values.

Good morning folks,

here is my solution much to late, but since I was looking for detecting outliers via PHP and could'nt find anything basic, I decided somehow smoothing a given dataset in a timeline of 24 h by simply moving a range of 5 items in a row through an unordered array and calculate the local standard deviation to detect the additive outliers.

The first function will simply calculate the average and deviation of a given array, where $col means the column with the values (sorry for the freegrades, this means that in an uncomplete dataset of 5 values you only have 4 freegrades - I don't know the exact english word for Freiheitsgrade):

function analytics_stat ($arr,$col,$freegrades = 0) {

// calculate average called mu
$mu = 0;
foreach ($arr as $row) {
    $mu += $row[$col];
}
$mu = $mu / count($arr);

// calculate empiric standard deviation called sigma
$sigma = 0;
foreach ($arr as $row) {
    $sigma += pow(($mu - $row[$col]),2);
}
$sigma = sqrt($sigma / (count($arr) - $freegrades));

return [$mu,$sigma];
}

Now its time for the core function, which will move through the given array and create a new array with the result. Margin means the factor to multiply the deviation with, since only one Sigma detects to many outliers, whereas more than 1.7 seems to high:

function analytics_detect_local_outliers ($arr,$col,$range,$margin = 1.0) {

$count = count($arr);
if ($count < $range) return false;

// the initial state of each value is NOT OUTLIER
$arr_result = [];
for ($i = 0;$i < $count;$i++) {
    $arr_result[$i] = false;
}

$max = $count - $range + 1;
for ($i = 0;$i < $max;$i++) {

    // calculate mu and sigma for current interval
    // remember that 5 values will determine the divisor 4 for sigma
    // since we only look at a part of the hole data set
    $stat = analytics_stat(array_slice($arr,$i,$range),$col,1);

    // a value in this interval counts, if it's found outside our defined sigma interval
    $range_max = $i + $range;
    for ($j = $i;$j < $range_max;$j++) {
        if (abs($arr[$j][$col] - $stat[0]) > $margin * $stat[1]) {
            $arr_result[$j] = true;

            // this would  be the place to add a counter to isolate
            // real outliers from sudden steps in our data set
        }
    }
}

return $arr_result;
}

And finally comes the test function with random values in an array with length 24. As for margin I was curious and choose the Golden Cut PHI = 1.618 ... since I really like this number and some Excel test results have led me to a margin of 1.7, above which outliers very rarelly were detected. The range of 5 is variable, but for me this was enough. So for every 5 values in a row there will be a calculation:

function test_outliers () {

// create 2 dimensional data array with items [hour,value]
$arr = [];
for ($i = 0;$i < 24;$i++) {
    $arr[$i] = [$i,rand(0,500)];
}

// set parameter for detection algorithm
$result = [];
$col = 1;
$range = 5;
$margin = 1.618;
$result = analytics_detect_local_outliers ($arr,$col,$range,$margin);

// display results
echo "<p style='font-size:8pt;'>";
for ($i = 0;$i < 24;$i++) {
    if ($result[$i]) echo "&diams;".$arr[$i][1]."&diams; "; else echo $arr[$i][1]." ";
}
echo "</p>";
}

After 20 calls of the test function I got these results:

417 140 372 131 449 26 192 222 320 349 94 147 201 ♦342♦ 123 16 15 ♦490♦ 78 190 ♦434♦ 27 3 276

379 440 198 135 22 461 208 376 286 ♦73♦ 331 358 341 14 112 190 110 266 350 232 265 ♦63♦ 90 94

228 ♦392♦ 130 134 170 ♦485♦ 17 463 13 326 47 439 430 151 268 172 342 445 477 ♦21♦ 421 440 219 95

88 121 292 255 ♦16♦ 223 244 109 127 231 370 16 93 379 218 87 ♦335♦ 150 84 181 25 280 15 406

85 252 310 122 188 302 ♦13♦ 439 254 414 423 216 456 321 85 61 215 7 297 337 204 210 106 149

345 411 308 360 308 346 ♦451♦ ♦77♦ 16 498 331 160 142 102 ♦496♦ 220 107 143 ♦241♦ 113 82 355 114 452

490 222 412 94 2 ♦480♦ 181 149 41 110 220 ♦477♦ 278 349 73 186 135 181 ♦39♦ 136 284 340 165 438

147 311 246 449 396 328 330 280 453 374 214 289 489 185 445 86 426 246 319 ♦30♦ 436 290 384 232

442 302 ♦436♦ 50 114 15 21 93 ♦376♦ 416 439 ♦222♦ 398 237 234 44 102 464 204 421 161 330 396 461

498 320 105 22 281 168 381 216 435 360 19 ♦402♦ 131 128 66 187 291 459 319 433 86 84 325 247

440 491 381 491 ♦22♦ 412 33 273 256 331 79 452 314 485 66 138 116 356 290 190 336 178 298 218

394 439 387 ♦80♦ 463 369 ♦104♦ 388 465 455 ♦246♦ 499 70 431 360 ♦22♦ 203 280 241 319 ♦34♦ 238 439 497

485 289 249 ♦416♦ 228 166 217 186 184 ♦356♦ 142 166 26 91 70 ♦466♦ 177 357 298 443 307 387 373 209

338 166 90 122 442 429 499 293 ♦41♦ 159 395 79 307 91 325 91 162 211 85 189 278 251 224 481

77 196 37 326 230 281 ♦73♦ 334 159 490 127 365 37 57 246 26 285 468 228 181 74 ♦455♦ 119 435

328 3 216 149 217 348 65 433 164 473 465 145 341 112 462 396 168 251 351 43 320 123 181 198

216 213 249 219 ♦29♦ 255 100 216 181 233 33 47 344 383 ♦94♦ 323 440 187 79 403 139 382 37 395

366 450 263 160 290 ♦126♦ 304 307 335 396 458 195 171 493 270 434 222 401 38 383 158 355 311 150

402 339 382 97 125 88 300 332 250 ♦86♦ 362 214 448 67 114 ♦354♦ 140 16 ♦354♦ 109 0 168 127 89

450 5 232 155 159 264 214 ♦416♦ 51 429 372 230 298 232 251 207 ♦322♦ 160 148 206 293 446 111 338

I hope, this will help anyone in the present or future. Greetings

P.S. To further improve this algorithm you may add a counter, which makes sure, that a certain value must for instance be found at least 2 times, that means in 2 different intervals or windows, before it is labeled as outlier. So a sudden jump of the following values does not make the first value the villain. Let me give you an example:

In 3,6,5,9,37,40,42,51,98,39,33,45 there is an obvious step from 9 to 37 and an isolated value 98. I would like to detect 98, but not 9 or 37. The first interval 3,6,5,9,37 would detect 37, the second interval 6,5,9,37,40 not. So we would not detect 37, since there is only one problematic interval or one match. Now it should be clear, that 98 counts in 5 intervals and is therefore an outlier. So lets declare a value an outlier, if it "counts" at least 2 times. Like so often we have to look closely the borders, since they have only one interval, and make for these values an exception.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.