Questions tagged [qubole]
Qubole Data Service (QDS) is cloud Big Data service running on an elastic Hadoop-based cluster
46
questions with no upvoted or accepted answers
3
votes
0
answers
408
views
Fetch all Column Statistics using Single Query Hive
I understand that all the column statistics can be computed for a Hive table using the command-
ANALYZE TABLE Table1 COMPUTE STATISTICS;
Then Specific column level stats can be fetched through the ...
2
votes
1
answer
291
views
need regexp_extract help, beginner
I have string column "49b8b35e-b62c-4a42-9d73-192d131d127a,03c8a7e0-5153-11ec-873a-0242ac11000a,eec8aee4-0500-4940-b319-15924cc2d248"
this string column has 3 values separate by ","...
2
votes
1
answer
43
views
Data comparisons in Qubole
I am very new to Qubole.We recently migrated Oracle ebiz data to Saleforce.We have both Ebiz and Salesforce data in the Qubole Data Lake.There are some discrepancies between Ebiz and Salesforce.What ...
2
votes
1
answer
5k
views
How to create external tables from parquet files in s3 using hive 1.2?
I have created an external table in Qubole(Hive) which reads parquet(compressed: snappy) files from s3, but on performing a SELECT * table_name I am getting null values for all columns except the ...
2
votes
1
answer
112
views
Debug failed shuffles in hadoop map reduces
I am seeing as the size of the input file increase failed shuffles increases and job complete time increases non linearly.
eg.
75GB took 1h
86GB took 5h
I also see average shuffle time increase 10 ...
2
votes
0
answers
819
views
Kafka Connect Hive Integration issue
I am using kafka connect for hive integration to create hive tables along with partitions on s3. After starting connect distributed process and making a post call to listen to a topic, as soon as ...
1
vote
0
answers
337
views
Insert overwrite doesn't delete all the old data files
We are trying to insert overwrite a hive table. Most of the times it's overwriting as expected, i.e deleting any old files and replace new files. We are seeing some inconsistencies with this behavior, ...
1
vote
1
answer
528
views
Retrieve value in an array of an array with struct
I have a column in Hive table with type:
array<array<struct<type:string,value:string,currency:string>>>
Here is the sample of data in the column:
[
[
{
"type":...
1
vote
0
answers
308
views
Query Qubole data in Python
I'm trying to query Qubole data in Python, but running into some issues. Below is my code:
from qds_sdk.qubole import Qubole
Qubole.configure(api_token="api_token", api_url="https://us....
1
vote
2
answers
102
views
How to connect UiPath to Qubole Hive cluster and run a query
One of the teams using RPA in my company wants to automate reporting that is run in Qubole - Hive environment. The initial approach is to unleash the robot to log in to Okta, then Workbench in Qubole, ...
1
vote
1
answer
201
views
Spark Submit Default Command line options
How can we change the parameters in Spark Submit Default Command line options in Qubole.
Though there is a option to override the values if needed under "Spark Submit Command Line Options" but this ...
1
vote
1
answer
548
views
How to create hive external table with avro file on qubole?
Can someone point in the doc to create external table on qubole base on avro files?
CREATE TABLE my_table_name
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS ...
1
vote
1
answer
66
views
Get Qubole data row wise using java
Am trying to run a hive query using Qubole SDK. Though am able to get the desired result as string, in order to better process it, am looking to access this row-wise. Something like a list of java ...
1
vote
1
answer
73
views
Recommendation on Performance optimization for SQL code
I have a code in Qubole that's taking almost 3 hours to execute. I am looking for some recommendations to decrease the code execution time.
WITH
-- Get latest date - 10 days before as day
d
AS (
...
1
vote
1
answer
252
views
Syncing Qubole HIve table to Snowflake with Struct field
I have a table like following Qubole:
use dm;
CREATE EXTERNAL TABLE IF NOT EXISTS fact (
id string,
fact_attr struct<
attr1 : String,
attr2 : String
>
)
STORED AS ...
1
vote
2
answers
153
views
Different results when distinct count by different time periods
I am trying to get a count of unique visitors. I first checked it by total without separating it by anytime frame.
Main table (big data table sample):
+-----------+----+-------+
|theDateTime|vD | ...
1
vote
0
answers
111
views
IN and NOT IN HiveQL
I am new to HiveQL and is IN and NOT IN supported in it? Especially when using Qubole?
Here is my query:
SELECT DISTINCT vId FROM table1
WHERE d.columnOne = "123"
AND NOT d.columnTwo
AND timestamp ...
1
vote
1
answer
1k
views
pyspark job on qubole fails with "Retrying exception reading mapper output"
I have a pyspark job running via qubole which fails with the following error.
Qubole > Shell Command failed, exit code unknown
Qubole > 2016-12-03 17:36:53,097 ERROR shellcli.py:231 - run - ...
1
vote
1
answer
476
views
How do I optimize my hive query for finding Sum of Count of Records from multiple tables
I’ve to generate a report that will give me the sum of the counts from tables A, B and C for events that have been stored using Hive and my S3 buckets have been partitioned by Organization_id
For eg:
...
0
votes
0
answers
16
views
Extracting json field from float in Hive using dataset
Quick one guys.
I am facing an issue while querying a float JSON Column, as it returns the following error:
"Error while compiling statement: FAILED: SemanticException [Error 10014]: line 11:5 ...
0
votes
0
answers
8
views
Connection issues with Qubole
How do I read data from s3n/data directly from AWS. At the moment I can trace my path to only data under s3 but not s3n.
I'm having this connection issues below when reading data from Qubole into ...
0
votes
1
answer
268
views
Presto Pivoting Data
I am really new to Presto and having trouble pivoting data in it.
The method I am using is the following:
select
distinct location_id,
case when role_group = 'IT' then employee_number end as ...
0
votes
2
answers
290
views
How to get Python in Qubole to save CSV and TXT files to Azure data lake?
I have Qubole connected to Azure data lake, and I can start a spark cluster, and run PySpark on it. However, I can't save any native Python output, like text files or CSVs. I can't save anything other ...
0
votes
1
answer
414
views
How to change the timeout value when running commands on QDS
I've a spark-submit command that calls my python script. The code runs more than 36 hours, however because of the QDS timeout limit of 36 hours my command gets killed after 36 hours.
Can someone help ...
0
votes
1
answer
257
views
Logging and Debuging on Qubole
How does one log on Qubole/access logs from spark on Qubole? The setup I have:
java library (JAR)
Zeppelin Notebook (Scala), simply calling a method from the library
Spark, Yarn cluster
Log4j2 used ...
0
votes
3
answers
82
views
How to query table partitions list using
I need to programmatically query Qubole for the list of partitions for a Hive table. I can do this by calling the correct API endpoint as described here, but I would like to use the qds-sdj-java ...
0
votes
1
answer
106
views
How to upgrade Python version on Qubole?
The current version on Qubole is 3.5.3, and some packages, like PyMC3 and future XGBoost need higher versions.
How do I upgrade? And would that affect other clusters' settings?
error message
0
votes
1
answer
266
views
Unable to write or read from S3 bucket with Default AWS KMS encryption enabled
I am unable to read or write into a Default AWS KMS encrypted bucket without using the following configuration on my Qubole cluster
fs.s3a.server-side-encryption-algorithm=SSE-KMS
fs.s3a.server-side-...
0
votes
1
answer
204
views
Qubole Kinesis Connector for Spark structured streaming throws an error
We are using Qubole Kinesis Connector (jar) for Spark structured streaming. This used to work fine but suddenly, it is throwing an error "S3 filesystem not found".
We could use the KCL but we need ...
0
votes
2
answers
335
views
Running Scala jobs in Scheduler
My job runs fine in my notebook, but when I copy and paste the script into the Spark Scala scheduled job, I run into errors like "script.scala:15: error: not found: value sqlContext".
What do I need ...
0
votes
1
answer
76
views
PySpark Machine Learning on Wide Data in Qubole
I have a large dataset, with roughly 250 features, that I would like to use in a gradient-boosted trees classifier. I have millions of observations, but I'm having trouble getting the model to work ...
0
votes
1
answer
84
views
Setting up AWS Glue to crawl Qubole
Currently I work with Qubole to access Hive data. I've added metadata from several databases, and want to add all the Hive metadata to AWS Glue. Is this possible? Any help is appreciated.
0
votes
1
answer
105
views
Scale plot size of matplotlib plots in Qubole Notebook
Is there a possibility of increasing the size of the plot plotted using z.showplot() in qubole notebooks.
import matplotlib as plt
plt.figure()
plt.bar(pandas_df_hr_sg[:]['hour'],pandas_df_hr_sg[:]['...
0
votes
1
answer
106
views
Performance analysis using Sparklens of Spark Streaming Application
I am trying to get performance analysis of a spark streaming application using sparklens. It is giving results like this
Executor count 1 ( 80%) estimated time 01m 29s and estimated cluster ...
0
votes
0
answers
818
views
How to fix 'Malformed class name' error in Spark Scala?
In Qubole notebook I am trying to get certain string from API response. It seems to be working just fine for sample data but fails when I use the full set. Spark version: 2.3.1; Scala version: 2.11; ...
0
votes
1
answer
44
views
retrieve size of data copied with hadoop distcp
I am running a hadoop distcp command as below:
hadoop distcp src-loc target-loc
I want to know the size of the data copied by running this command.
I am planning to run the command on Qubole.
Any ...
0
votes
1
answer
987
views
Big files causing shuffle error in hadoop map reduce
I am seeing the following error when I try to process big file like size > 35GB files, but doesn't happen when I try less big file like size < 10GB .
App > Error: org.apache.hadoop.mapreduce....
0
votes
1
answer
124
views
Get correct value from array in Hive QL
I have a Wrapped Array and want to only get the corresponding value struct when I query with LATERAL VIEW EXPLODE.
SAMPLE STRUCTURE:
COLUMNNAME: theARRAY
WrappedArray([null,theVal,valTags,[123,...
0
votes
1
answer
97
views
Amazon s3Exception bad request and location constraint in hadoop s3a
Does location constraint require extra permission policy for hadoop s3a?
I am seeing
Exception in thread "main" com.qubole.com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: ...
0
votes
1
answer
182
views
Set partition location in Qubole metastore using Spark
How to set partition location for my Hive table in Qubole metastore?
I know that this is MySQL DB, but how to access to it and pass a SQL script with a fix using Spark?
UPD: The issue is that ALTER ...
0
votes
0
answers
120
views
Container Packing in YARN
Qubole has implemented Container Packing in YARN for cloud deployments to reduce infrastructure cost, is there any similar implementation available in open source world?
0
votes
1
answer
121
views
Qubole: How can I download scheduler result in python?
Like title, I managed myself download the Qubole result using the query id in python, however, is there a method that I can download the result using scheduler job ID instead of query ID?
Thanks.
0
votes
1
answer
93
views
HDFS Data Export error: could only be replicated to 0 nodes, instead of 1
After succussefully running an Hive Query when Data Export to MySQL starts in Qubole, I am getting the following error (in my log file):
org.apache.hadoop.ipc.RemoteException: java.io.IOException: ...
-1
votes
1
answer
135
views
trying to execute s3-sqs qubole connector for spark structured streaming
I am trying to follow, https://github.com/qubole/s3-sqs-connector and trying to load the connector but seems like the connector is not available on maven and while generating the buiold manually the ...
-1
votes
1
answer
82
views
Can I write an HTML script and pass information from the script to a cell on Qubole?
Is it possible to write an HTML script and have the user interact on the HTML script and pass the data back to the zeppelin cell and have it rerun the data passed back?
Thank you!
Update:
Have some ...
-2
votes
1
answer
153
views
Error running presto query on kinesis
I am getting following error while running presto query on kinesis.
Query 20151031_142753_00003_xxxxx failed: Rate exceeded for stream xxxxx under account xxxxxx. (Service: AmazonKinesis; Status ...