4

I'll formulate a simple problem that I'd like to solve with machine learning (in R or similar platforms): my algorithm takes 3 parameters (a,b,c), and returns a score s in range [0,1]. The parameters are all categorical: a has 3 options, b has 4, and c has 10. Therefore my dataset has 3 * 4 * 10 = 120 cases. High scores are desirable (close to 1), low scores are not (close to 0). Let's treat the algorihm as a black box, taking a,b,c and returning a s.

The dataset looks like this:

a, b, c, s
------------------
a1, b1, c1, 0.223
a1, b1, c2, 0.454
...

If I plot the density of the s for each parameter, I get very wide distributions, in which some cases perform very well (s > .8 ), others badly (s < .2 ).

If I look at the cases where s is very high, I can't see any clear pattern. Parameter values that overall perform badly can perform very well in combination with specific parameters, and vice versa.

To measure how well a specific value performs (e.g. a1), I compute the median:

median( mydataset[ a == a1]$s )

For example, median(a1)=.5, median(b3)=.9, but when I combine them, I get a lower result s(a_1,b_3)= .3. On the other hand, median(a2)=.3, median(b1)=.4, but s(a2,b1)= .7.

Given that there aren't parameter values that perform always well, I guess I should look for combinations (of 2 parameters) that seem to perform well together, in a statistically significant way (i.e. excluding outliers that happen to have very high scores). In other words, I want to obtain a policy to make the optimal parameter choice, e.g. the best performing combinations are (a1,b3), (a2,b1), etc.

Now, I guess that this is an optimisation problem that can be solved using machine learning.

What standard techniques would you recommend in this context?

EDIT: somebody suggested a linear programming solution with glpk, but I don't understand how to apply linear programming to this problem.

4
  • The most standard technique for this question is Linear Regression. You may predict the value for specific parameters; in more general - to get the function that on your 3 parameters gives you maximum value
    – Anton
    Mar 22, 2012 at 12:42
  • When I run a regression I get some coefficients for a1,a2,etc, and a R squared. Not sure how they tell me the top performing values.
    – Mulone
    Mar 22, 2012 at 16:56
  • Since there are only 120 cases, you could compute the expected score conditional on the values of a, b and c and see which combination performs best. You would need a large amount of data to prevent overfitting, though. To get a rough idea of significance of your results, you could compute a p-value for each conditional expectation. If they are all sufficiently low, you're done. If not, then you may want to look at some kind of smoothing (e.g. shrinking towards the average score). Mar 23, 2012 at 9:47
  • If you use a linear regression, you are making very strong assumptions about the interactions between parameter values (i.e. that the only interactions are additive). Mar 23, 2012 at 9:49

1 Answer 1

1

The most standard technique for this question is Linear Regression. You may predict the value for specific parameters; in more general - to get the function that on your 3 parameters gives you maximum value

2
  • When I run a regression I get some coefficients for a1,a2,etc, and a R squared. Not sure how they tell me the top performing values.
    – Mulone
    Mar 22, 2012 at 16:56
  • 1
    If you use a linear regression, you are making very strong assumptions about the interactions between parameter values (i.e. that the only interactions are additive). Mar 23, 2012 at 9:49

Not the answer you're looking for? Browse other questions tagged or ask your own question.