Questions tagged [qubole]
Qubole Data Service (QDS) is cloud Big Data service running on an elastic Hadoop-based cluster
85
questions
0
votes
0
answers
61
views
Convert column in presto from epoch to date [duplicate]
I tried this but that didn't work.
cast(from_unixtime('1532568232662880')) as date
Any other ideas?
0
votes
1
answer
97
views
Amazon s3Exception bad request and location constraint in hadoop s3a
Does location constraint require extra permission policy for hadoop s3a?
I am seeing
Exception in thread "main" com.qubole.com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: ...
1
vote
1
answer
1k
views
Import csv file into Qubole
I am using qubole to run presto queries.
I need to upload a csv file into my query but cannot figure out how to do this.
Does anyone have any experience with this?
For more details, I am under ...
0
votes
1
answer
2k
views
How do I get the value without the square brackets [duplicate]
I have created a dataframe using Scala and Spark SQL. I wanted the first value from the table but I am getting it inside of square brackets []. Can I just get the value without the brackets?
Code:
...
0
votes
0
answers
66
views
select a table from a database in R [duplicate]
I am using dbplyr to select a table from a remote database using Rstudio. I connected with Spark in the server using livy. It shows me the databases I have but when I try to access one of the tables ...
2
votes
2
answers
5k
views
Fixing java.lang.NoSuchMethodError: com.amazonaws.util.StringUtils.trim
Consider the following error:
2018-07-12 22:46:36,087 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: com.amazonaws.util.StringUtils.trim(Ljava/...
3
votes
0
answers
408
views
Fetch all Column Statistics using Single Query Hive
I understand that all the column statistics can be computed for a Hive table using the command-
ANALYZE TABLE Table1 COMPUTE STATISTICS;
Then Specific column level stats can be fetched through the ...
0
votes
1
answer
311
views
Get Not Null Values in Wrapped Array
I have a Wrapped Array and want to only get the Non Null values when I query with LATERAL VIEW EXPLODE. I also tried IS NOT NULL but that does not return anything.
SAMPLE STRUCTURE:
COLUMNNAME: ...
1
vote
0
answers
111
views
IN and NOT IN HiveQL
I am new to HiveQL and is IN and NOT IN supported in it? Especially when using Qubole?
Here is my query:
SELECT DISTINCT vId FROM table1
WHERE d.columnOne = "123"
AND NOT d.columnTwo
AND timestamp ...
1
vote
1
answer
1k
views
UDF to generate JSON string behaving inconsistently
I'm trying to generate a JSON string to store a variable number of history records in a single STRING column. The code works on all of my small tests, but fails (no error, just no data) when run on ...
0
votes
1
answer
182
views
Set partition location in Qubole metastore using Spark
How to set partition location for my Hive table in Qubole metastore?
I know that this is MySQL DB, but how to access to it and pass a SQL script with a fix using Spark?
UPD: The issue is that ALTER ...
0
votes
0
answers
120
views
Container Packing in YARN
Qubole has implemented Container Packing in YARN for cloud deployments to reduce infrastructure cost, is there any similar implementation available in open source world?
1
vote
1
answer
144
views
Run Tensorflow in Qubole
I am trying to train LSTM using Spark python Notebook in Qubole. When I try to fit model, I received below error.
I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions ...
0
votes
1
answer
121
views
Qubole: How can I download scheduler result in python?
Like title, I managed myself download the Qubole result using the query id in python, however, is there a method that I can download the result using scheduler job ID instead of query ID?
Thanks.
0
votes
1
answer
412
views
unable to connect ms sql server from Presto in Qubole
I am using Qubole Data Service on Microsoft Azure. I have created Presto Cluster in Qubole. I want to connect to MS SQL Server from Presto to read data from MS SQL Server. I have created sqlserver ...
2
votes
1
answer
259
views
java.io.FileNotFound exception while writing to apache spark in qubole
I have a code in apache spark 1.6.3 running on qubole which writes data to multiple tables(parquet format) on s3. At the time of writing to tables I keep getting java.io.FileNotFound exception.
I am ...
1
vote
1
answer
238
views
How to select records from week days?
I have hive table which contain daily records. I want to select record from week days. So i use bellow hive query to do it. I'm using QUBOLE API to do this.
SELECT hour(pickup_time),
...
2
votes
0
answers
819
views
Kafka Connect Hive Integration issue
I am using kafka connect for hive integration to create hive tables along with partitions on s3. After starting connect distributed process and making a post call to listen to a topic, as soon as ...
5
votes
1
answer
2k
views
How to kill hadoop job gracefully/intercept `hadoop job -kill`
My Java application runs on mapper and creates child processes using Qubole API. Application stores child qubole queryIDs. I need to intercept kill signal and shutdown child processes before exit. ...
0
votes
1
answer
151
views
Comparing one day worth of data from S3 buckets faster
Consider 2 data flows below
1. Front End Box ----> S3 Bucket-1
2. Front End Box ----> Kafka --> Storm ---> S3 Bucket-2
The logs from the boxes are being transferred to S3 buckets. The ...
0
votes
1
answer
1k
views
How to query data from gz file of Amazon S3 using Qubole Hive query?
I need get specific data from gz.
how to write the sql?
can I just sql as table database?:
Select * from gz_File_Name where key = 'keyname' limit 10.
but it always turn back with an error.
1
vote
1
answer
329
views
AWS S3 access issue when using qubole/streamx on AWS EMR
I am using qubole/streamx as a kafka sink connector to consume data in kafka and store them in AWS S3.
I created a user in AIM and permission is AmazonS3FullAccess. Then set key ID and key in hdfs-...
0
votes
1
answer
102
views
Adding results of Hadoop job to Hive Table
I have a Map-only job that processes a large text file. Each line is analyzed and categorized. MultipleOutputs are used to output each category into separate files. Eventually all the data gets ...
0
votes
2
answers
1k
views
Inserting local csv to a Hive table from Qubole
I have a csv on my local machine, and I access Hive through Qubole web console. I am trying to upload the csv as a new table, but couldn't figure out. I have tried the following:
LOAD DATA LOCAL ...
1
vote
1
answer
1k
views
pyspark job on qubole fails with "Retrying exception reading mapper output"
I have a pyspark job running via qubole which fails with the following error.
Qubole > Shell Command failed, exit code unknown
Qubole > 2016-12-03 17:36:53,097 ERROR shellcli.py:231 - run - ...
4
votes
1
answer
3k
views
Divide Spark DataFrame data into separate files
I have the following DataFrame input from a s3 file and need to transform the data into the following desired output. I am using Spark version 1.5.1 with Scala, but could change to Spark with Python. ...
1
vote
1
answer
476
views
How do I optimize my hive query for finding Sum of Count of Records from multiple tables
I’ve to generate a report that will give me the sum of the counts from tables A, B and C for events that have been stored using Hive and my S3 buckets have been partitioned by Organization_id
For eg:
...
0
votes
1
answer
93
views
HDFS Data Export error: could only be replicated to 0 nodes, instead of 1
After succussefully running an Hive Query when Data Export to MySQL starts in Qubole, I am getting the following error (in my log file):
org.apache.hadoop.ipc.RemoteException: java.io.IOException: ...
0
votes
2
answers
474
views
Is there an api to send notifications based on job outputs?
I know there are api to configure the notification when a job is failed or finished.
But what if, say, I run a hive query that count the number of rows in a table. If the returned result is zero I ...
1
vote
1
answer
114
views
Unable to create table in Qubole similar to mysql
I want to create a external table in Qubole similar to a table created in Mysql. Query for create table in mysql is:
CREATE TABLE `mytable` (
`id` varchar(50) NOT NULL,
`v_count` int(11) DEFAULT ...
-2
votes
1
answer
153
views
Error running presto query on kinesis
I am getting following error while running presto query on kinesis.
Query 20151031_142753_00003_xxxxx failed: Rate exceeded for stream xxxxx under account xxxxxx. (Service: AmazonKinesis; Status ...
2
votes
1
answer
209
views
Median value from table with number:count format
Given a table
+------------+-----------+
| Number | Count |
+------------+-----------+
| 0 | 7 |
+------------+-----------+
| 1 | 1 |
+------------+----------...
3
votes
1
answer
960
views
Insert into ElasticSearch using Hive/Qubole
I am trying to insert data into elastic search from a hive table.
CREATE EXTERNAL TABLE IF NOT EXISTS es_temp_table (
dt STRING,
text STRING
)
STORED BY 'org....
1
vote
2
answers
2k
views
Autoscaling EMR- is it required? Should I just use EC2? Should I just use Qubole?
In order to reduce the time for provisioning, we've decided to keep up a dedicated EMR cluster with 5 instances (we expect to need about 5). In case we need more, we think we'll need to implement some ...
6
votes
1
answer
4k
views
Stratified Sampling in Hive
The following returns a 10% sample of the A and X columns stratified by the values of X.
select A, X from(
select A,
count(*) over (partition by X) as cnt,
rank() over (partition ...