Questions tagged [sparkling-water]

Sparkling Water integrates H2O's fast scalable machine learning engine with Spark.

Filter by
Sorted by
Tagged with
0 votes
1 answer
22 views

Facing issues with H2O MOJO deployment using sparkling water on Google Dataproc

I am trying to migrate a model from Hadoop to GCP. Model MOJO will not be retrained. I am running the model in Dataproc using Airflow spark submit. Source data format matches with Hadoop source and ...
trougc's user avatar
  • 329
0 votes
0 answers
20 views

Error When trying to run sparkling-shell on windows

I have installed sparkling water for apache spark 3.2.3 and trying to run sparkling-shell command on cmd I also installed "spark3.2.3-bin-hadoop3.2" from the apache spark archive which is ...
Saideva Sathvik Ravula's user avatar
0 votes
0 answers
19 views

H2OAutoML.fit() fails with "Wrong FS" exception when keepBinaryModels=True on Windows

Training works fine with keepBinaryModels=False, but when set True I get: Traceback (most recent call last): File "C:\spark\spark-3.2.4-bin-hadoop2.7\python\pyspark\sql\utils.py", line 117,...
8forty's user avatar
  • 545
0 votes
1 answer
40 views

Get Stage Results from H2O Sparkling Water model

I am looking to create a confidence interval for one of my model's outputs and I need to get the model outputs before the link function is applied. From what I've read, it seems like I am interested ...
omoshiroiii's user avatar
0 votes
0 answers
30 views

h2o sparkling: Error reading MOJO JSON: Object not supported:

I'm running the Sparkling Water automl example (https://docs.h2o.ai/sparkling-water/3.3/latest-stable/doc/ml/sw_automl.html) on a local spark install on Windows (added code at the top setting up the ...
8forty's user avatar
  • 545
0 votes
2 answers
67 views

How do I save H2O Sparkling Water models to disk

I have a PySpark code to train an H2o DRF model. I need to save this model to disk and then load it. from pysparkling.ml import H2ODRF drf = H2ODRF(featuresCols = predictors, labelCol =...
Mahdi's user avatar
  • 787
0 votes
1 answer
160 views

How to get the leader feature importances in h2o automl pysparkling water

i am using spark standalone cluster and running h2o pysparkling in it. I am unable to find the function for getting the leader feature importances. please help Code: import pandas as pd from pyspark....
Priyadutt Bhatt's user avatar
0 votes
1 answer
101 views

Getting error while setting spark.ext.h2o.backend.cluster.mode=external in pysparkling standalone cluster

Code: import pandas as pd from pyspark.sql import SparkSession from pysparkling import * import h2o from pysparkling.ml import H2OAutoML spark = SparkSession.builder.appName('SparkApplication')....
Priyadutt Bhatt's user avatar
0 votes
1 answer
213 views

I am getting error while defining H2OContext in python spark script

Code: from pyspark.sql import SparkSession from pysparkling import * hc = H2OContext.getOrCreate() I am using spark standalone cluster 3.2.1 and try to initiate H2OContext in python file. while ...
Priyadutt Bhatt's user avatar
0 votes
1 answer
166 views

h2o-pysparkling-2.4 and Glue Jobs with: {"error":"TypeError: 'JavaPackage' object is not callable","errorType":"EXECUTION_FAILURE"}

I am try to using pysparkling.ml.H2OMOJOModel for predict a spark dataframe using a MOJO model trained with h2o==3.32.0.2 in AWS Glue Jobs, how ever a got the error: TypeError: 'JavaPackage' object is ...
MaxReis86's user avatar
1 vote
1 answer
78 views

Import POJO model with Sparkling Water (Scala)

I am trying to import a POJO model into Sparkling Water. I am currently importing the model by compiling it using: javac -cp /opt/bitnami/commons/pojo.jar -J-Xmx2g -J-XX:MaxPermSize=256m /opt/bitnami/...
Joaquín Romero Moreno's user avatar
0 votes
1 answer
250 views

How to specify the H2O version in Sparkling Water?

In a Databricks notebook, I am trying to load an H2O model that was trained for H2O version 3.30.1.3. I have installed the version of Sparkling Water which corresponds to the Spark version used for ...
James Adams's user avatar
  • 8,449
1 vote
2 answers
82 views

Are H2O.ai products affected by log4shell vulnerability?

My question is if Open Source H2O-3, Open Source Sparkling Water and Driverless AI are affected by CVE-2021-44228 and CVE-2021-45046.
Michal's user avatar
  • 437
1 vote
0 answers
125 views

ERROR Instrumentation: java.lang.ClassNotFoundException: ai.h2o.sparkling.ml.models.H2OGBMMOJOModel

I am working on pysparkling in Databricks. I have built a model with pyspark transformers and h2o pysparkling algorithm. When I log the model on to mlflow and deploy in from a job cluster, I get the ...
Pradeep Ganesan's user avatar
2 votes
0 answers
227 views

How to shade packages inside a fat jar depdency

I've an SBT project that depends on "com.google.cloud.bigdataoss" % "gcs-connector" % "hadoop3-2.2.2" which is bringing a recent version of google-api-services-storage. ...
bachr's user avatar
  • 5,780
0 votes
1 answer
197 views

Unable to access H2O Flow using H2O_connection_url

I am using both H2O and Sparkling Water on Amazon Clusters. I have been using Qubole and have been able to access the Flow UI on that platform. I am currently testing Databricks and Sagemaker, but I ...
David Comfort's user avatar
1 vote
1 answer
534 views

Error in creating H2OContext in databricks using pysparkling

I am using spark version 2.4.4 and h2o-pysparkling-2.4 on the databricks and running following code h2oConf = H2OConf().set('spark.sql.autoBroadcastJoinThreshold', '-1') hc = H2OContext.getOrCreate(...
user9482910's user avatar
2 votes
1 answer
71 views

Error H2O cluster should be of size 3 but is 2

I'm trying to run H2O SW on Kubernetes using the steps in the documentation. I launch a test SW app $ bin/spark-submit \ --master k8s://$KUBERNETES_ENDPOINT \ --deploy-mode cluster \ --class ai.h2o....
bachr's user avatar
  • 5,780
0 votes
0 answers
116 views

com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT at cross_validation_metrics_summary

I'm using the H2ODRF and H2OGridSearch models to create a random forest pipeline with random discrete grid search hyper-parameter optimization. However, when I set nfolds to any number greater than 1 ...
charming_chatterjee's user avatar
1 vote
1 answer
395 views

Error when importing Sparkling Water (H2O) pipeline in Apache Spark: py4j.protocol.Py4JError

I recently created a PySpark pipeline using Sparkling Water's AutoML in the last stage (very similar to https://github.com/h2oai/sparkling-water/blob/master/py/examples/pipelines/...
luis_ferreira223's user avatar
2 votes
2 answers
309 views

H2OGridSearch H2OGBM pyspark: NullPointerException in extractH2OParameters

I'm trying to run a grid search for Gradient Boosting Machine in pyspark with H2O Sparkling Water. Produced a reproducible example with the famous iris dataset. from pysparkling import H2OContext, ...
lrnzcig's user avatar
  • 3,867
0 votes
1 answer
249 views

Is it possible to use MLFlow and H2o.ai sparkling water in a Scala based project?

I'm solving a Scala data science problem in Intellij using maven. I noticed that MLFlow spark (https://mvnrepository.com/artifact/org.mlflow/mlflow-spark/1.5.0) is dependent on scala 2.12 while h2o.ai ...
proselotis's user avatar
0 votes
0 answers
200 views

Predict function giving error for XGBoost but running for GBM in H2O

I am making a classification model using H2O in python. I am able to build a GBM model and make predictions on training and test dataset whereas when I build an XGBoost model and try to make ...
Vishal Gupta's user avatar
3 votes
0 answers
236 views

H2O Sparkling Water AutoML not working properly in Spark Scala. Exception: Ease restrictions on setMaxModels or setMaxRuntimeSecs

I'm looking into H2O sparkling water AutoML using scala. I'm running it on my laptop on localhost. Even though I'm not adding any restrictions on H2OAutoML() class using setMaxModels or ...
Bhushan Gosavi's user avatar
0 votes
1 answer
140 views

Disable H2O Flow UI

I'm planning to use H2O with sparkling water (vesion 3.26.0.2) in production. Thus it is reuired to disable Flow UI that is available via port 54321. I was unable to find a configuration property to ...
Gandras's user avatar
  • 23
0 votes
2 answers
129 views

Run coxph model for large data set with 300 columns( 6 GB ) in H2o sparkling water

We are trying to run coxph model using h2o,Rsparkling for large data set with 6 GB with 300 columns, whatever the configuration we take for spark, we are getting memory issues. As per h2o, we should ...
Divya M's user avatar
  • 43
1 vote
1 answer
318 views

Get model metrics from MOJO model in H2O

I have a MOJO model that I want to explore for model metrics (rmse,roc, etc.) I understand all model metrics are available for a binary model, but I want to get these metrics from a MOJO model. ...
curios's user avatar
  • 11
0 votes
0 answers
1k views

NameError: name 'H2OContext' is not defined in Pycham

I'm trying to run sparkling-water from pycham, I have install the spark and sparkling-water-2.4.13.but getting error "NameError: name 'H2OContext' is not defined". The same code is working from cli. ...
Dev Tanwar's user avatar
1 vote
1 answer
145 views

How do I exclude algorithms from H2O AutoML in Sparkling Water by using Scala

I have to exclude some algorithms from AutoMl model. I am trying this to exclude algorithms but it fails. buildSpecHopper_1.build_models.exclude_algos = Array(Algo.DeepLearning,Algo.GLM) But it ...
Amandeep singh saini's user avatar
0 votes
1 answer
111 views

Does H2O Sparkling Water allow for Online-Training with Kafka as Streaming Source

I'm currently experimenting with the possibilities of Sparkling-Water. There are a few possible Use-Cases including Data-Munging in H2O/Spark, Model Building and Offline-Training and Online Stream ...
dnks23's user avatar
  • 359
0 votes
1 answer
83 views

Create H2O Sparkling-Water app in IntelliJ

I want to set up a Sparkling-Water app in IntelliJ. I found the droplet for a project at: Sparkling-Water-Droplet But this has not been touched for a year and I was wondering whether there is a more ...
dnks23's user avatar
  • 359
0 votes
1 answer
236 views

Frame upload/creation on H2O external backend hangs from python/pyspark

I'm experiencing an issue where the h2o.H2OFrame([1,2,3]) command is creating a frame within h2o on an internal backend, but not on an external backend. Instead, the connection is not terminating (the ...
0_0's user avatar
  • 574
0 votes
2 answers
158 views

How change the column type numeric to enum in sparkling water using scala?

I have to change numeric columns to Enum type of h2o frame in sparkling water using Scala and how to print schema of h2o frame.
Amandeep singh saini's user avatar
0 votes
1 answer
353 views

rsparkling as_h2o_frame does not work: java.lang.OutOfMemoryError: GC overhead limit exceeded

I first import a dataset from csv to Spark, do some transformation in Spark, and then try to convert it into H2O Frame. Here's my code: library(rsparkling) library(h2o) library(dplyr) library(...
Catiger3331's user avatar
0 votes
1 answer
167 views

From h2o, is there a way to export N folder cross validation results into a dataframe?

I am using H2O sparking water to built GBM model. I know we can view the N folder cross validation results using code below: gbm_model.model_performance(xval = True) But is there a way to save each ...
Gavin's user avatar
  • 1,411
2 votes
1 answer
250 views

Excluding interecept in H2O (python and R) produces non-zero coefficient for intercept anyway

I am trying to use an H2O library in both Python and R to produce a GLM without an intercept included. Unfortunately, it does not appear to be working. The results are completely off, the intercept ...
Alex's user avatar
  • 43
0 votes
1 answer
1k views

Create partial dependence plot using H2O in spark?

I am trying to create partial dependent plot using the following code rf_pdp = rf_model .partial_plot(data = htest, cols = ['var1', 'var2', 'var3'], plot=True) rf_pdp it runs without error and ...
Gavin's user avatar
  • 1,411
0 votes
1 answer
330 views

How to make H2OGridSearch for H2OGradientBoostingEstimator repeatable (Reproducibility) in spark environment?

I am using the following code to run GBM in Sparkling Water. I have set up the seed and score_each_iteration, but every time, it still generates different results when I check the AUC even though I ...
Gavin's user avatar
  • 1,411
1 vote
1 answer
128 views

H2O sparkling water - DNN mini_batch_size parameter

I'm currently running Spark 2.3.0 with sparkling-water 2.3.1. I found the documentation of the underlying H2O library by looking at the changelog that links to this. So apparently it uses H2O 3.18. By ...
se7entyse7en's user avatar
  • 4,330
0 votes
1 answer
186 views

h2o deeplearning error when specifying nfolds for cross validation

has this issue been resolved by now? I encounter the same error message. Usecase: I am doing binary classification using h2o's deeplearning() function. Below, I provide randomly generated data the ...
KSA's user avatar
  • 3
0 votes
1 answer
450 views

Training H2O Stacked Ensemble Models using exported Mojo and Binary Models

I am trying to build stacked ensemble models using H2O Java APIs. For this, I trained 2 models A GBM Model A DRF Model I exported these models in both Mojo and Binary format. For exporting models,...
AdityaK01's user avatar
0 votes
1 answer
78 views

How to embed custom plots in the H2O flow UI in Scala or R?

After some investigation, I have found out that the sparkling water H2O flow UI has a very limited set of plots - just Box plots, and distributions, for data visualization in Scala. But if I want to ...
user3243499's user avatar
  • 2,963
0 votes
1 answer
41 views

How to make plots for data exploration in sparkling waters h2oflow UI?

I am trying to explore a sample dataset by visualizing the data using plots like Box plots, scatter plots, histograms, etc. Unfortunately, I not able to find any commands even in the H2o ...
user3243499's user avatar
  • 2,963
1 vote
0 answers
126 views

pysparkling H2OConf interfering with my application log

Here is my code: from pysparkling import H2OConf #commenting this line makes it work import logging logging.basicConfig(filename='my_log.log',level=logging.INFO) logging.info('test') I cannot get ...
Tiberiu's user avatar
  • 1,000
0 votes
0 answers
176 views

Unable to start sparkling shell in Windows

I am running Windows 10 and using Scala 2.11 and spark version 2.2.1. Spark_home is also configured and path is also added to it's bin and I can start spark-shell directly from command prompt from any ...
user3243499's user avatar
  • 2,963
0 votes
1 answer
487 views

not able to create H2OContext in Databricks- using pysparkling

I am not able to create H2OContext in Spark Databricks- using pysparkling. It is giving the following error. Code:from pysparkling import * Code:import h2o Code:h2oConf = H2OConf(spark) Code:h2oConf....
Marvania Mehul's user avatar
1 vote
1 answer
2k views

How can I embed H2o in a Java application?

I am trying start embedded H2o in a Java application and train a model. However I don't get what exactly explained in the documentation (http://docs.h2o.ai/h2o/latest-stable/h2o-docs/faq/java.html). ...
Esildor's user avatar
  • 169
1 vote
0 answers
67 views

Productionizing Spark Pipeline [duplicate]

Using Sparkling Water/H2o v. 2.3 for prediction. I am trying to export a spark pipeline model containing H2o model. Scoring will need to be done on a java based platform. Please suggest best method ...
Arun Lakhotia's user avatar
0 votes
1 answer
79 views

Configuring pysparkling logger with custom format

I'm trying to productionize Python Sparkling Water application and I want to unify logging formats from my app, Spark and H2O. I was able to modify log4j.properties in Spark home and achieve it with ...
omikron's user avatar
  • 2,755
1 vote
2 answers
485 views

Sparkling water won't start on Spark on Google DataProc

I'm trying to use H2O Sparkling Water on Google DataProc. I've successfully run Sparkling Water on a standalone Spark, and now moved on to use it on DataProc. Initially, I got an error about spark....
cybermaggedon's user avatar