2

I have a set of weighted features for machine learning. I'd like to reduce the feature set and just use those with a very large or very small weight.

So given below image of sorted weights, I'd only like to use the features that have weights above the higher or below the lower yellow line.

Sorted features; yellow lines are the desired cut-off points (thresholds)

What I'm looking for is some kind of slope change detection so I can discard all the features until the first/last slope coefficient increase/decrease.

While I (think I) know how to code this myself (with first and second numerical derivatives), I'm interested in any established methods. Perhaps there's some statistic or index that computes something like that, or anything I can use from SciPy?

Edit: At the moment, I'm using 1.8*positive.std() as positive and 1.8*negative.std() as negative threshold (fast and simple), but I'm not mathematician enough to determine how robust this is. I don't think it is, though. ⍨

enter image description here

1 Answer 1

0

If the data are (approximately) Gaussian distributed, then just using a multiple of the standard deviation is sensible.

If you are worried about heavier tails, then you may want to base your analysis on order statistics.

  • Since you've plotted it, I'll assume you're willing to sort all of the data.
  • Let N be the number of data points in your sample.
  • Let x[i] be the i'th value in the sorted list of values.
  • Then 0.5( x[int( 0.8413*N)]-x[int(0.1587*N)]) is an estimate of the standard deviation which is more robust against outliers. This estimate of the std can be used as you indicated above. (The magic numbers above are the fraction of data that are less than [mean+1sigma] and [mean-1sigma] respectively).

  • There are also conditions where just keeping the highest 10% and lowest 10% would be sensible as well; and these cutoffs are easily computed if you have the sorted data on hand.

These are somewhat ad hoc approaches based on the content of your question. The general sense of what you're trying to do is (a form of) anomaly detection, and you can probably do a better job of it if you're careful in defining/estimating what the shape of the distribution is near the middle, so that you can tell when the features are getting anomalous.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.