2

I'm using r package of random forest to predict the distances between pairs of proteins based on their amino acid sequence, the main interest is the proteins that are close (has smaller distance). my training dataset consist of 10k pair of proteins and the actual distance between them. however, very few pairs of protein (less than 0.2%) has small distances between them, and the problem is that the trained random forest became very accurate in predicting the distance between proteins with large distances and very bad for proteins that have small distances between them. I tried to down-sample the proteins with the large distances in my training data, but the results are still not good. I'm more interested in close proteins (those pairs who have small distance between them). there is a very clear signal of over-fitting since my training accuracy is 78 and my testing accuracy is 51% any suggestions are highly appreciated

1
  • You have a case of imbalanced data. 0.2% is insignificant so the model (random forest) ignores them. If your test set contains more data with small distances you should move them to training set and use cross-validation for testing accuracy. Likely this won't be enough so resampling of small distance samples is next step (search 'imbalanced data resampling')
    – topchef
    Mar 21, 2013 at 13:48

2 Answers 2

1

A couple suggestions:

1) Look at GBM's from the gbm package.

2) Create more features to help the RF understand what drives distance.

3) Plot errors vs individual variables to look for what is driving relationships. (ggplot2 is great for this especially using the colour and size options.)

4) You could also assign 1 or 0 to y-variables based on distance (ie if distance < x; set to 1 / if distance >= x; set to 0). Once you have two classes you can use the strata argument in RF to create uniformly balanced samples and see what variables are driving the difference in distance using the importance() and varImpPlot() functions of RF.

5) Try using log of distance-related variables. RF is usually pretty good about compensating for non-linearity but it can't hurt to try.

My guess is that #2 is where you want to spend your time though it is also the hardest and requires the most thought.

1

I think what might help you given your problem is Synthetic Minority Over-Sampling Technique for Regression (SMOTER). There is some research on this topic. However, it remains less explored than its classification counterpart (SMOTE), as you have likely encountered.

I might suggest the paper cited below depending on how interested you are in understanding it from a research perspective. I really appreciated the introduction of Gaussian Noise in generating the synthetic observations.

If you're more interested in a practical solution, the first author has an R implementation available on her Github page. https://github.com/paobranco/SMOGN-LIDTA17

If Python is more of your persuasion, I recently distributed an entirely Pythonic implementation of the SMOGN algorithm that is now available and currently being unit tested. https://github.com/nickkunz/smogn

Branco, P., Torgo, L., Ribeiro, R. (2017). "SMOGN: A Pre-Processing Approach for Imbalanced Regression". Proceedings of Machine Learning Research, 74:36-50.http://proceedings.mlr.press/v74/branco17a/branco17a.pdf.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.