Questions tagged [sparkling-water]
Sparkling Water integrates H2O's fast scalable machine learning engine with Spark.
128
questions
0
votes
1
answer
22
views
Facing issues with H2O MOJO deployment using sparkling water on Google Dataproc
I am trying to migrate a model from Hadoop to GCP. Model MOJO will not be retrained. I am running the model in Dataproc using Airflow spark submit.
Source data format matches with Hadoop source and ...
0
votes
0
answers
20
views
Error When trying to run sparkling-shell on windows
I have installed sparkling water for apache spark 3.2.3 and trying to run sparkling-shell command on cmd
I also installed "spark3.2.3-bin-hadoop3.2" from the apache spark archive which is ...
0
votes
0
answers
19
views
H2OAutoML.fit() fails with "Wrong FS" exception when keepBinaryModels=True on Windows
Training works fine with keepBinaryModels=False, but when set True I get:
Traceback (most recent call last):
File "C:\spark\spark-3.2.4-bin-hadoop2.7\python\pyspark\sql\utils.py", line 117,...
0
votes
1
answer
40
views
Get Stage Results from H2O Sparkling Water model
I am looking to create a confidence interval for one of my model's outputs and I need to get the model outputs before the link function is applied. From what I've read, it seems like I am interested ...
0
votes
0
answers
30
views
h2o sparkling: Error reading MOJO JSON: Object not supported:
I'm running the Sparkling Water automl example (https://docs.h2o.ai/sparkling-water/3.3/latest-stable/doc/ml/sw_automl.html) on a local spark install on Windows (added code at the top setting up the ...
0
votes
2
answers
67
views
How do I save H2O Sparkling Water models to disk
I have a PySpark code to train an H2o DRF model. I need to save this model to disk and then load it.
from pysparkling.ml import H2ODRF
drf = H2ODRF(featuresCols = predictors,
labelCol =...
0
votes
1
answer
160
views
How to get the leader feature importances in h2o automl pysparkling water
i am using spark standalone cluster and running h2o pysparkling in it.
I am unable to find the function for getting the leader feature importances. please help
Code:
import pandas as pd
from pyspark....
0
votes
1
answer
101
views
Getting error while setting spark.ext.h2o.backend.cluster.mode=external in pysparkling standalone cluster
Code:
import pandas as pd
from pyspark.sql import SparkSession
from pysparkling import *
import h2o
from pysparkling.ml import H2OAutoML
spark = SparkSession.builder.appName('SparkApplication')....
0
votes
1
answer
213
views
I am getting error while defining H2OContext in python spark script
Code:
from pyspark.sql import SparkSession
from pysparkling import *
hc = H2OContext.getOrCreate()
I am using spark standalone cluster 3.2.1 and try to initiate H2OContext in python file. while ...
0
votes
1
answer
166
views
h2o-pysparkling-2.4 and Glue Jobs with: {"error":"TypeError: 'JavaPackage' object is not callable","errorType":"EXECUTION_FAILURE"}
I am try to using pysparkling.ml.H2OMOJOModel for predict a spark dataframe using a MOJO model trained with h2o==3.32.0.2 in AWS Glue Jobs, how ever a got the error: TypeError: 'JavaPackage' object is ...
1
vote
1
answer
78
views
Import POJO model with Sparkling Water (Scala)
I am trying to import a POJO model into Sparkling Water. I am currently importing the model by compiling it using:
javac -cp /opt/bitnami/commons/pojo.jar -J-Xmx2g -J-XX:MaxPermSize=256m /opt/bitnami/...
0
votes
1
answer
250
views
How to specify the H2O version in Sparkling Water?
In a Databricks notebook, I am trying to load an H2O model that was trained for H2O version 3.30.1.3.
I have installed the version of Sparkling Water which corresponds to the Spark version used for ...
1
vote
2
answers
82
views
Are H2O.ai products affected by log4shell vulnerability?
My question is if Open Source H2O-3, Open Source Sparkling Water and Driverless AI are affected by CVE-2021-44228 and CVE-2021-45046.
1
vote
0
answers
125
views
ERROR Instrumentation: java.lang.ClassNotFoundException: ai.h2o.sparkling.ml.models.H2OGBMMOJOModel
I am working on pysparkling in Databricks. I have built a model with pyspark transformers and h2o pysparkling algorithm. When I log the model on to mlflow and deploy in from a job cluster, I get the ...
2
votes
0
answers
227
views
How to shade packages inside a fat jar depdency
I've an SBT project that depends on
"com.google.cloud.bigdataoss" % "gcs-connector" % "hadoop3-2.2.2"
which is bringing a recent version of google-api-services-storage.
...
0
votes
1
answer
197
views
Unable to access H2O Flow using H2O_connection_url
I am using both H2O and Sparkling Water on Amazon Clusters. I have been using Qubole and have been able to access the Flow UI on that platform. I am currently testing Databricks and Sagemaker, but I ...
1
vote
1
answer
534
views
Error in creating H2OContext in databricks using pysparkling
I am using spark version 2.4.4 and h2o-pysparkling-2.4 on the databricks and running following code
h2oConf = H2OConf().set('spark.sql.autoBroadcastJoinThreshold', '-1')
hc = H2OContext.getOrCreate(...
2
votes
1
answer
71
views
Error H2O cluster should be of size 3 but is 2
I'm trying to run H2O SW on Kubernetes using the steps in the documentation.
I launch a test SW app
$ bin/spark-submit \
--master k8s://$KUBERNETES_ENDPOINT \
--deploy-mode cluster \
--class ai.h2o....
0
votes
0
answers
116
views
com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT at cross_validation_metrics_summary
I'm using the H2ODRF and H2OGridSearch models to create a random forest pipeline with random discrete grid search hyper-parameter optimization. However, when I set nfolds to any number greater than 1 ...
1
vote
1
answer
395
views
Error when importing Sparkling Water (H2O) pipeline in Apache Spark: py4j.protocol.Py4JError
I recently created a PySpark pipeline using Sparkling Water's AutoML in the last stage (very similar to https://github.com/h2oai/sparkling-water/blob/master/py/examples/pipelines/...
2
votes
2
answers
309
views
H2OGridSearch H2OGBM pyspark: NullPointerException in extractH2OParameters
I'm trying to run a grid search for Gradient Boosting Machine in pyspark with H2O Sparkling Water.
Produced a reproducible example with the famous iris dataset.
from pysparkling import H2OContext, ...
0
votes
1
answer
249
views
Is it possible to use MLFlow and H2o.ai sparkling water in a Scala based project?
I'm solving a Scala data science problem in Intellij using maven. I noticed that MLFlow spark (https://mvnrepository.com/artifact/org.mlflow/mlflow-spark/1.5.0) is dependent on scala 2.12 while h2o.ai ...
0
votes
0
answers
200
views
Predict function giving error for XGBoost but running for GBM in H2O
I am making a classification model using H2O in python. I am able to build a GBM model and make predictions on training and test dataset whereas when I build an XGBoost model and try to make ...
3
votes
0
answers
236
views
H2O Sparkling Water AutoML not working properly in Spark Scala. Exception: Ease restrictions on setMaxModels or setMaxRuntimeSecs
I'm looking into H2O sparkling water AutoML using scala. I'm running it on my laptop on localhost. Even though I'm not adding any restrictions on H2OAutoML() class using setMaxModels or ...
0
votes
1
answer
140
views
Disable H2O Flow UI
I'm planning to use H2O with sparkling water (vesion 3.26.0.2) in production.
Thus it is reuired to disable Flow UI that is available via port 54321.
I was unable to find a configuration property to ...
0
votes
2
answers
129
views
Run coxph model for large data set with 300 columns( 6 GB ) in H2o sparkling water
We are trying to run coxph model using h2o,Rsparkling for large data set with 6 GB with 300 columns, whatever the configuration we take for spark, we are getting memory issues.
As per h2o, we should ...
1
vote
1
answer
318
views
Get model metrics from MOJO model in H2O
I have a MOJO model that I want to explore for model metrics (rmse,roc, etc.) I understand all model metrics are available for a binary model, but I want to get these metrics from a MOJO model.
...
0
votes
0
answers
1k
views
NameError: name 'H2OContext' is not defined in Pycham
I'm trying to run sparkling-water from pycham, I have install the spark and sparkling-water-2.4.13.but getting error "NameError: name 'H2OContext' is not defined".
The same code is working from cli.
...
1
vote
1
answer
145
views
How do I exclude algorithms from H2O AutoML in Sparkling Water by using Scala
I have to exclude some algorithms from AutoMl model.
I am trying this to exclude algorithms but it fails.
buildSpecHopper_1.build_models.exclude_algos = Array(Algo.DeepLearning,Algo.GLM)
But it ...
0
votes
1
answer
111
views
Does H2O Sparkling Water allow for Online-Training with Kafka as Streaming Source
I'm currently experimenting with the possibilities of Sparkling-Water. There are a few possible Use-Cases including Data-Munging in H2O/Spark, Model Building and Offline-Training and Online Stream ...
0
votes
1
answer
83
views
Create H2O Sparkling-Water app in IntelliJ
I want to set up a Sparkling-Water app in IntelliJ. I found the droplet for a project at: Sparkling-Water-Droplet
But this has not been touched for a year and I was wondering whether there is a more ...
0
votes
1
answer
236
views
Frame upload/creation on H2O external backend hangs from python/pyspark
I'm experiencing an issue where the h2o.H2OFrame([1,2,3]) command is creating a frame within h2o on an internal backend, but not on an external backend. Instead, the connection is not terminating (the ...
0
votes
2
answers
158
views
How change the column type numeric to enum in sparkling water using scala?
I have to change numeric columns to Enum type of h2o frame in sparkling water using Scala and how to print schema of h2o frame.
0
votes
1
answer
353
views
rsparkling as_h2o_frame does not work: java.lang.OutOfMemoryError: GC overhead limit exceeded
I first import a dataset from csv to Spark, do some transformation in Spark, and then try to convert it into H2O Frame. Here's my code:
library(rsparkling)
library(h2o)
library(dplyr)
library(...
0
votes
1
answer
167
views
From h2o, is there a way to export N folder cross validation results into a dataframe?
I am using H2O sparking water to built GBM model. I know we can view the N folder cross validation results using code below:
gbm_model.model_performance(xval = True)
But is there a way to save each ...
2
votes
1
answer
250
views
Excluding interecept in H2O (python and R) produces non-zero coefficient for intercept anyway
I am trying to use an H2O library in both Python and R to produce a GLM without an intercept included. Unfortunately, it does not appear to be working. The results are completely off, the intercept ...
0
votes
1
answer
1k
views
Create partial dependence plot using H2O in spark?
I am trying to create partial dependent plot using the following code
rf_pdp = rf_model .partial_plot(data = htest, cols = ['var1', 'var2', 'var3'], plot=True)
rf_pdp
it runs without error and ...
0
votes
1
answer
330
views
How to make H2OGridSearch for H2OGradientBoostingEstimator repeatable (Reproducibility) in spark environment?
I am using the following code to run GBM in Sparkling Water. I have set up the seed and score_each_iteration, but every time, it still generates different results when I check the AUC even though I ...
1
vote
1
answer
128
views
H2O sparkling water - DNN mini_batch_size parameter
I'm currently running Spark 2.3.0 with sparkling-water 2.3.1. I found the documentation of the underlying H2O library by looking at the changelog that links to this. So apparently it uses H2O 3.18.
By ...
0
votes
1
answer
186
views
h2o deeplearning error when specifying nfolds for cross validation
has this issue been resolved by now? I encounter the same error message.
Usecase: I am doing binary classification using h2o's deeplearning() function. Below, I provide randomly generated data the ...
0
votes
1
answer
450
views
Training H2O Stacked Ensemble Models using exported Mojo and Binary Models
I am trying to build stacked ensemble models using H2O Java APIs.
For this, I trained 2 models
A GBM Model
A DRF Model
I exported these models in both Mojo and Binary format. For exporting models,...
0
votes
1
answer
78
views
How to embed custom plots in the H2O flow UI in Scala or R?
After some investigation, I have found out that the sparkling water H2O flow UI has a very limited set of plots - just Box plots, and distributions, for data visualization in Scala.
But if I want to ...
0
votes
1
answer
41
views
How to make plots for data exploration in sparkling waters h2oflow UI?
I am trying to explore a sample dataset by visualizing the data using plots like Box plots, scatter plots, histograms, etc.
Unfortunately, I not able to find any commands even in the H2o ...
1
vote
0
answers
126
views
pysparkling H2OConf interfering with my application log
Here is my code:
from pysparkling import H2OConf #commenting this line makes it work
import logging
logging.basicConfig(filename='my_log.log',level=logging.INFO)
logging.info('test')
I cannot get ...
0
votes
0
answers
176
views
Unable to start sparkling shell in Windows
I am running Windows 10 and using Scala 2.11 and spark version 2.2.1. Spark_home is also configured and path is also added to it's bin and I can start spark-shell directly from command prompt from any ...
0
votes
1
answer
487
views
not able to create H2OContext in Databricks- using pysparkling
I am not able to create H2OContext in Spark Databricks- using pysparkling. It is giving the following error.
Code:from pysparkling import *
Code:import h2o
Code:h2oConf = H2OConf(spark)
Code:h2oConf....
1
vote
1
answer
2k
views
How can I embed H2o in a Java application?
I am trying start embedded H2o in a Java application and train a model. However I don't get what exactly explained in the documentation (http://docs.h2o.ai/h2o/latest-stable/h2o-docs/faq/java.html). ...
1
vote
0
answers
67
views
Productionizing Spark Pipeline [duplicate]
Using Sparkling Water/H2o v. 2.3 for prediction. I am trying to export a spark pipeline model containing H2o model.
Scoring will need to be done on a java based platform.
Please suggest best method ...
0
votes
1
answer
79
views
Configuring pysparkling logger with custom format
I'm trying to productionize Python Sparkling Water application and I want to unify logging formats from my app, Spark and H2O.
I was able to modify log4j.properties in Spark home and achieve it with ...
1
vote
2
answers
485
views
Sparkling water won't start on Spark on Google DataProc
I'm trying to use H2O Sparkling Water on Google DataProc. I've successfully run Sparkling Water on a standalone Spark, and now moved on to use it on DataProc. Initially, I got an error about spark....