Questions tagged [qubole]
Qubole Data Service (QDS) is cloud Big Data service running on an elastic Hadoop-based cluster
85
questions
6
votes
1
answer
4k
views
Stratified Sampling in Hive
The following returns a 10% sample of the A and X columns stratified by the values of X.
select A, X from(
select A,
count(*) over (partition by X) as cnt,
rank() over (partition ...
5
votes
1
answer
2k
views
How to kill hadoop job gracefully/intercept `hadoop job -kill`
My Java application runs on mapper and creates child processes using Qubole API. Application stores child qubole queryIDs. I need to intercept kill signal and shutdown child processes before exit. ...
4
votes
1
answer
3k
views
Divide Spark DataFrame data into separate files
I have the following DataFrame input from a s3 file and need to transform the data into the following desired output. I am using Spark version 1.5.1 with Scala, but could change to Spark with Python. ...
3
votes
1
answer
960
views
Insert into ElasticSearch using Hive/Qubole
I am trying to insert data into elastic search from a hive table.
CREATE EXTERNAL TABLE IF NOT EXISTS es_temp_table (
dt STRING,
text STRING
)
STORED BY 'org....
3
votes
0
answers
408
views
Fetch all Column Statistics using Single Query Hive
I understand that all the column statistics can be computed for a Hive table using the command-
ANALYZE TABLE Table1 COMPUTE STATISTICS;
Then Specific column level stats can be fetched through the ...
2
votes
1
answer
5k
views
Pyspark Logging: Printing information at the wrong log level
Thanks for your time!
I'd like to create and print legible summaries of my (hefty) data to my output when debugging my code, but stop creating and printing those summaries once finished to speed ...
2
votes
2
answers
5k
views
Fixing java.lang.NoSuchMethodError: com.amazonaws.util.StringUtils.trim
Consider the following error:
2018-07-12 22:46:36,087 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: com.amazonaws.util.StringUtils.trim(Ljava/...
2
votes
1
answer
259
views
java.io.FileNotFound exception while writing to apache spark in qubole
I have a code in apache spark 1.6.3 running on qubole which writes data to multiple tables(parquet format) on s3. At the time of writing to tables I keep getting java.io.FileNotFound exception.
I am ...
2
votes
1
answer
209
views
Median value from table with number:count format
Given a table
+------------+-----------+
| Number | Count |
+------------+-----------+
| 0 | 7 |
+------------+-----------+
| 1 | 1 |
+------------+----------...
2
votes
1
answer
348
views
How do you write a presto query to split a string into its own column
Trying to splint a string into multiple columns in qubole using presto query.
{"field0":[{"startdate":"2022-07-13","lastnightdate":"2022-07-16","...
2
votes
1
answer
291
views
need regexp_extract help, beginner
I have string column "49b8b35e-b62c-4a42-9d73-192d131d127a,03c8a7e0-5153-11ec-873a-0242ac11000a,eec8aee4-0500-4940-b319-15924cc2d248"
this string column has 3 values separate by ","...
2
votes
1
answer
43
views
Data comparisons in Qubole
I am very new to Qubole.We recently migrated Oracle ebiz data to Saleforce.We have both Ebiz and Salesforce data in the Qubole Data Lake.There are some discrepancies between Ebiz and Salesforce.What ...
2
votes
1
answer
5k
views
How to create external tables from parquet files in s3 using hive 1.2?
I have created an external table in Qubole(Hive) which reads parquet(compressed: snappy) files from s3, but on performing a SELECT * table_name I am getting null values for all columns except the ...
2
votes
1
answer
112
views
Debug failed shuffles in hadoop map reduces
I am seeing as the size of the input file increase failed shuffles increases and job complete time increases non linearly.
eg.
75GB took 1h
86GB took 5h
I also see average shuffle time increase 10 ...
2
votes
0
answers
819
views
Kafka Connect Hive Integration issue
I am using kafka connect for hive integration to create hive tables along with partitions on s3. After starting connect distributed process and making a post call to listen to a topic, as soon as ...
1
vote
2
answers
2k
views
Autoscaling EMR- is it required? Should I just use EC2? Should I just use Qubole?
In order to reduce the time for provisioning, we've decided to keep up a dedicated EMR cluster with 5 instances (we expect to need about 5). In case we need more, we think we'll need to implement some ...
1
vote
2
answers
261
views
Result-set inconsistency between hive and hive-llap
we are using Hive 3.1.x clusters on HDI 4.0, with 1 being LLAP and another Just HIVE.
we've created a managed tables on both the clusters with the row count being 272409.
Before merge on both ...
1
vote
2
answers
163
views
Implement case class inside a class
I am using the below code to run in Qubole Notebook and the code is running successfully.
case class cls_Sch(Id:String, Name:String)
class myClass {
implicit val sparkSession = org.apache.spark....
1
vote
1
answer
584
views
Extracting json field from string in Hive using dataset
I am trying a very basic hive query. I am trying to extract a json field from a dataset but I always get
\N
for the json field, however some_string comes okay
Here is my query :
WITH dataset AS ...
1
vote
1
answer
1k
views
Import csv file into Qubole
I am using qubole to run presto queries.
I need to upload a csv file into my query but cannot figure out how to do this.
Does anyone have any experience with this?
For more details, I am under ...
1
vote
1
answer
144
views
Run Tensorflow in Qubole
I am trying to train LSTM using Spark python Notebook in Qubole. When I try to fit model, I received below error.
I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions ...
1
vote
1
answer
238
views
How to select records from week days?
I have hive table which contain daily records. I want to select record from week days. So i use bellow hive query to do it. I'm using QUBOLE API to do this.
SELECT hour(pickup_time),
...
1
vote
1
answer
329
views
AWS S3 access issue when using qubole/streamx on AWS EMR
I am using qubole/streamx as a kafka sink connector to consume data in kafka and store them in AWS S3.
I created a user in AIM and permission is AmazonS3FullAccess. Then set key ID and key in hdfs-...
1
vote
1
answer
114
views
Unable to create table in Qubole similar to mysql
I want to create a external table in Qubole similar to a table created in Mysql. Query for create table in mysql is:
CREATE TABLE `mytable` (
`id` varchar(50) NOT NULL,
`v_count` int(11) DEFAULT ...
1
vote
0
answers
337
views
Insert overwrite doesn't delete all the old data files
We are trying to insert overwrite a hive table. Most of the times it's overwriting as expected, i.e deleting any old files and replace new files. We are seeing some inconsistencies with this behavior, ...
1
vote
1
answer
528
views
Retrieve value in an array of an array with struct
I have a column in Hive table with type:
array<array<struct<type:string,value:string,currency:string>>>
Here is the sample of data in the column:
[
[
{
"type":...
1
vote
0
answers
308
views
Query Qubole data in Python
I'm trying to query Qubole data in Python, but running into some issues. Below is my code:
from qds_sdk.qubole import Qubole
Qubole.configure(api_token="api_token", api_url="https://us....
1
vote
1
answer
76
views
Exclude records with certain values in Qubole
Using Qubole
I have
Table A (columns in json parsed...)
ID Recommendation Decision
1 GOOD GOOD
2 BAD BAD
2 GOOD BAD
3 GOOD BAD
4 ...
1
vote
2
answers
102
views
How to connect UiPath to Qubole Hive cluster and run a query
One of the teams using RPA in my company wants to automate reporting that is run in Qubole - Hive environment. The initial approach is to unleash the robot to log in to Okta, then Workbench in Qubole, ...
1
vote
1
answer
201
views
Spark Submit Default Command line options
How can we change the parameters in Spark Submit Default Command line options in Qubole.
Though there is a option to override the values if needed under "Spark Submit Command Line Options" but this ...
1
vote
1
answer
548
views
How to create hive external table with avro file on qubole?
Can someone point in the doc to create external table on qubole base on avro files?
CREATE TABLE my_table_name
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS ...
1
vote
1
answer
66
views
Get Qubole data row wise using java
Am trying to run a hive query using Qubole SDK. Though am able to get the desired result as string, in order to better process it, am looking to access this row-wise. Something like a list of java ...
1
vote
1
answer
73
views
Recommendation on Performance optimization for SQL code
I have a code in Qubole that's taking almost 3 hours to execute. I am looking for some recommendations to decrease the code execution time.
WITH
-- Get latest date - 10 days before as day
d
AS (
...
1
vote
1
answer
252
views
Syncing Qubole HIve table to Snowflake with Struct field
I have a table like following Qubole:
use dm;
CREATE EXTERNAL TABLE IF NOT EXISTS fact (
id string,
fact_attr struct<
attr1 : String,
attr2 : String
>
)
STORED AS ...
1
vote
2
answers
153
views
Different results when distinct count by different time periods
I am trying to get a count of unique visitors. I first checked it by total without separating it by anytime frame.
Main table (big data table sample):
+-----------+----+-------+
|theDateTime|vD | ...
1
vote
0
answers
111
views
IN and NOT IN HiveQL
I am new to HiveQL and is IN and NOT IN supported in it? Especially when using Qubole?
Here is my query:
SELECT DISTINCT vId FROM table1
WHERE d.columnOne = "123"
AND NOT d.columnTwo
AND timestamp ...
1
vote
1
answer
1k
views
UDF to generate JSON string behaving inconsistently
I'm trying to generate a JSON string to store a variable number of history records in a single STRING column. The code works on all of my small tests, but fails (no error, just no data) when run on ...
1
vote
1
answer
1k
views
pyspark job on qubole fails with "Retrying exception reading mapper output"
I have a pyspark job running via qubole which fails with the following error.
Qubole > Shell Command failed, exit code unknown
Qubole > 2016-12-03 17:36:53,097 ERROR shellcli.py:231 - run - ...
1
vote
1
answer
476
views
How do I optimize my hive query for finding Sum of Count of Records from multiple tables
I’ve to generate a report that will give me the sum of the counts from tables A, B and C for events that have been stored using Hive and my S3 buckets have been partitioned by Organization_id
For eg:
...
0
votes
2
answers
59
views
Rest api in testdrive account?
Hi I am using Qubole trial version and it is test drive account
so I am not getting API Token from control panel my accounts tab in qubole
is there a way to access REST API's Now?
Thanks in Advance
0
votes
1
answer
412
views
unable to connect ms sql server from Presto in Qubole
I am using Qubole Data Service on Microsoft Azure. I have created Presto Cluster in Qubole. I want to connect to MS SQL Server from Presto to read data from MS SQL Server. I have created sqlserver ...
0
votes
1
answer
567
views
How to safely insert parameters into a SQL query and get the resulting query?
I have to use a non DBAPI-compliant library to interact with a database (qds_sdk for Qubole). This library only allows to send raw SQL queries without parameters. Thus I would like a SQL injection-...
0
votes
1
answer
580
views
Avoid pre-signed URL expiry when IAM role key rotates
In Airflow I have 2 tasks defined that run every day:
the first one creates a zip file and saves it in AWS under s3://{bucket-name}/foo/bar/{date}/archive.zip
the second one pre-signs that url (...
0
votes
1
answer
344
views
Qubole Presto datatype "Map" using the Like Operator
So I am trying to apply a simple like function for a Qubole query on Presto. For a string datatype I can simply do like '%United States of America%'.
However for the column I am trying to apply ...
0
votes
1
answer
46
views
In the new Analyze UI, how do I edit the title of my query?
In the new Qubole Analyze UI that came out recently, I cannot seem to find a way to change the title of a command. In the old interface, I could click on the command title and it would become an ...
0
votes
1
answer
311
views
Get Not Null Values in Wrapped Array
I have a Wrapped Array and want to only get the Non Null values when I query with LATERAL VIEW EXPLODE. I also tried IS NOT NULL but that does not return anything.
SAMPLE STRUCTURE:
COLUMNNAME: ...
0
votes
1
answer
1k
views
How to query data from gz file of Amazon S3 using Qubole Hive query?
I need get specific data from gz.
how to write the sql?
can I just sql as table database?:
Select * from gz_File_Name where key = 'keyname' limit 10.
but it always turn back with an error.
0
votes
2
answers
1k
views
Inserting local csv to a Hive table from Qubole
I have a csv on my local machine, and I access Hive through Qubole web console. I am trying to upload the csv as a new table, but couldn't figure out. I have tried the following:
LOAD DATA LOCAL ...
0
votes
2
answers
474
views
Is there an api to send notifications based on job outputs?
I know there are api to configure the notification when a job is failed or finished.
But what if, say, I run a hive query that count the number of rows in a table. If the returned result is zero I ...
0
votes
0
answers
16
views
Extracting json field from float in Hive using dataset
Quick one guys.
I am facing an issue while querying a float JSON Column, as it returns the following error:
"Error while compiling statement: FAILED: SemanticException [Error 10014]: line 11:5 ...