3

I have been trying to replicate the results from cross-validation from h2o by hand, but I'm puzzled on why the results are different. In my understanding, if a fold_column is specified, then cross-validation happens according to the groups defined within that column. Therefore, for fold 1, the model should train on folds 2, 3, 4 and 5.

set.seed(1)
iris$fold <- sample(1:5, nrow(iris), replace = TRUE)

fit_cv <- h2o.glm(x = c("Species", "Petal.Width", "Petal.Length"),
                  y = c("Sepal.Length"),
                  training_frame = as.h2o(iris),
                  fold_column = "fold", 
                  alpha = 0.2, lambda = 0.1, seed=1,
                  keep_cross_validation_models = TRUE)

This outputs for fold 1 a training MAE of:

fit_fold1 <-  h2o.cross_validation_models(fit_cv)[[1]]

> fit_fold1@model$training_metrics@metrics$mae
[1] 0.3035526

However, if fitted by hand, the training MAE is (slightly) different:

dd_fold1_fit <- iris[iris$fold != 1,]

fit_fold1_direct <- h2o.glm(x = c("Species", "Petal.Width", "Petal.Length"),
                                y = c("Sepal.Length"),
                                training_frame = as.h2o(dd_fold1_fit),
                                alpha = 0.2, lambda = 0.1, 
                                seed=1)
    
fit_fold1_direct@model$training_metrics@metrics$mae
[1] 0.3004132

This is a minimal example, the issue is that with real data that I am working with the difference is larger, and I can't replicate the cross-validation results.

Any idea on what the explanation could be?

1
  • I have just noticed the same thing. The h2o.cross_validation_predictions for a given fold do not even match the h2o.cross_validation_models
    – chipsin
    Sep 23, 2022 at 13:21

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.