Questions tagged [druid]
Druid is a column-oriented open-source distributed data store written in Java.
547
questions
0
votes
0
answers
22
views
Add "total" column as a dimension in Druid queries
Is it possible to aggregate the following dataset:
property_name
metric_value
A
20
B
40
C
40
null
100
Into:
property_name
metric_value
total
A
20
100
B
40
100
C
40
100
My goal is to ...
0
votes
1
answer
19
views
Is there a Druid SQL Dialect?
I'm using Spring Data JPA with druid.
I added dependencies spring-boot-starter-data-jpa and avatica-core to use druid JDBC Driver with spring data jpa in my project.
<dependency>
...
0
votes
1
answer
27
views
Druid SQL vs Druid Native Query
I have some question about Druid Query.
According to official document, there are two query language which are Druid query and Native query.
In my use, I feel more comfortable to use Druid Sql because ...
0
votes
0
answers
24
views
Druid indexing task not picking up by middlemanager
I have a druid cluster with version 0.23 and deployed on kubernetes using druid operator. When I submit indexing task the task doesn't go to middle manager node instead it goes to coordinator node. ...
0
votes
1
answer
42
views
spark-submit doesn't work with avtiva jdbc driver
I am trying to write a spark job with simple query operation connecting to Druid. According to Druid document,I am using avatica jdbc driver to connect to Druid.
Sample code:
object SimpleApp {
def ...
0
votes
0
answers
25
views
Schema registry in Druid | Using Kafka as an ingestion source
I’m trying to ingest the data to druid using Kafka as an ingestion source. The Kafka topics which I’m using are having schema registry URLs with a specific username and password associated with the ...
0
votes
1
answer
32
views
How to write log and data in Druid Deep Storage in AWS S3
We have a druid cluster setup and now i am trying to write the indexing-logs and data into S3 deep storage.
Following are the details
druid.storage.type=s3
druid.storage.bucket=bucket-name
druid....
0
votes
1
answer
31
views
Unable to inject data from postgresql to druid (deployed using helm)
I deployed druid using helm from repository using commands from https://github.com/apache/druid/tree/master/helm/druid and got it deployed successfully but when I created a task with following spec
{
...
0
votes
0
answers
39
views
Ingest data from mongodb to druid
I have an NLP model that saves data to a mongo database. All this with Java.
My problem is that I want to make a report of this data, in my company they use druid and aws lambda to generate reports, ...
0
votes
2
answers
35
views
How to group by single field and return more values together
I'm starting to use apache druid but having some difficult to run native queries (and some SQL too).
1- Is it possible to groupBy a single column while also returning more channels?
2- How could I ...
0
votes
0
answers
19
views
Connect to Druid using Zapr java client with username and password
Currently, we are using druid in our application using the Zapr java client but we do not have a username and password in Druid.
Now in another environment druid is having username and password. We ...
0
votes
0
answers
31
views
Different default behaviors of skipping empty buckets in timeseries queries by Druid native query and Druid SQL
According to the document of Apache Druid about native query
Timeseries queries normally fill empty interior time buckets with zeroes.
For example, if you issue a "day" granularity ...
0
votes
1
answer
21
views
Druid sql functions analogue to Prometheus 'rate' and 'histogram_quantile'
I wrote an application and deployed a Prometheus, that scrapes metrics from the app and stores it in Druid (I'm forced to use this database in production for metrics). Now I want to make a dashboard ...
0
votes
0
answers
31
views
Is there a way to bypass problem of package split in java automatic modules when we're not in charge of included code
I have a custom library for creating requests to Apache Druid database which I would like to turn into java modules.
My library includes dependencies:
org.apache.druid:druid-processing:0.22.1
org....
0
votes
1
answer
23
views
druid-kinesis-indexing-service giving unparseable characters
I am switching our ingestion from using tranquility to use druid-kinesis-indexing-service. However, when I connect to the data it is showing lines with correct json sandwiched between unparseable ...
0
votes
2
answers
147
views
Migrate Kafka Topic to new Cluster (and impact on Druid)
I am ingesting data into Druid from Kafka's topic. Now I want to migrate my Kafka Topic to the new Kafka Cluster. What are the possible ways to do this without duplication of data and without downtime?...
0
votes
1
answer
65
views
Druid generate missing records
I have a data table in druid and which has missing rows and I want to fill them by generating the missing timestamps and adding the precedent row value.
This is the table in druid :
| __time ...
0
votes
0
answers
63
views
Create new Druid Datasource in Apache Superset
I'm creating a Druid Datasource in Apache Superset.
I see 'Is Hidden' checkbox, but when I check or uncheck it, I don't see any difference.
Can anyone explain to me What 'Is Hidden' checkbox means?
0
votes
1
answer
174
views
Druid Group by Day of Week and Hour of Day
I am trying to find how to group by day of week, and also hour of day in Druid. I am relying on Druid documentation here: https://druid.apache.org/docs/latest/querying/sql.html, and all the following ...
0
votes
1
answer
152
views
Druid SQL: filter on result of expression
I have HTTP access log data in a Druid data source, and I want to see access patterns based on certain identifiers in the URL path. I wrote this query, and it works fine:
select regexp_extract(path, '/...
0
votes
1
answer
88
views
How to get Quantile/median values in pydruid
My goal is to query the median value of column height in my druid datasource. I was able to use other aggregations like count and count distinct values. Here's my query so far:
group = query.groupby(
...
0
votes
0
answers
68
views
Apache Druid Appending Segment without dropping or summing it
I have three JSON files with the same timestamp but different values to upload to the Druid. I want to upload them separately with the same segment granularity. However, it drops the existing segment ...
0
votes
2
answers
277
views
So slow Apache Druid Query
Currently i'm working with an Apache Druid Warehouse that stores near of 300 Millions of rows and have a size of 44GB. We are developing a Flask API that uses Gunicorn and Celery to develop SQL query ...
0
votes
1
answer
68
views
Amazon kinesis injestion on Apache Druid
Hi After loading data iam getting unhealthy superviser as status in ingestion
when check the logs errors are
"state": "UNHEALTHY_SUPERVISOR",
"detailedState": "...
0
votes
1
answer
98
views
Can we access historical data stored in druid
Assuming if we are storing historical data in Apache Druid can we from nodejs access them, including filtering, searching ..etc
Is there any api in druid so that we can write rest api's like we write ...
0
votes
1
answer
274
views
Apache Druid can store null value for number
I write spec to collect data from kafka.
But I know that Druid will auto convert null value of type double, long to 0.0 and 0. '
But I want to store null value instead of 0.
So what should I do when ...
0
votes
0
answers
168
views
How to tell Druid to skip bad kafka records during ingestion with schema registry?
Long story short: Druid fails to ingest corrupted kafka record. How can I skip it?
I configured Druid to ingest data from kafka with schema_registry successfully. Everything works as expected until ...
-1
votes
1
answer
245
views
How to change the druid admin default password?
In Druid I am using default basic authenticator
My config look like this
druid.auth.authenticatorChain=["MyBasicMetadataAuthenticator"]
druid.auth.authenticator.MyBasicMetadataAuthenticator....
0
votes
1
answer
152
views
Druid Cluster going into Restricted Mode
We have a Druid Cluster with the following specs
3X Coordinators & Overlords - m5.2xlarge
6X Middle Managers(Ingest nodes with 5 slots) - m5d.4xlarge
8X Historical - i3.4xlarge
2X Router & ...
0
votes
1
answer
63
views
Druid Groupby: druid's groupby considering only 1000 records per group to aggregate Average
I was calculating average of each group. Noticed that average it is calculating only for 1000 values per group instead of considering the complete data points.
Trying to calculate Average time taken ...
0
votes
0
answers
139
views
Getting an error when creating a external table for a druid data source in hive
I have a separate druid server. I am trying to create a druid data source from hive beeline shell using following query
SET hive.druid.broker.address.default=ip:8082;
CREATE EXTERNAL TABLE ...
0
votes
1
answer
72
views
Druid how to quick flush data from tables
I want quick remove data form Druid. Currently, I'm marking rows as unused and then run kill.
This procedure take 2 min for 5x rows. It is possible to do it faster, i.e. by reset druid or something?
0
votes
2
answers
235
views
How to get results of multiple aggregations in a single druid query?
Say I have the following table named t_student_details:
Name Age Marks Sport City ........ (multiple columns)
====== ===== ======= ======= ======
Jason 11 45 ...
0
votes
0
answers
46
views
Is it enough to upgrade Java JDK/JRE if still compiling with an old java.version?
I'm using an open-source project which still uses Java 8. It supports Java 11, but the pom.xml locks the compiled byte-code version to 8:
<maven.compiler.source>1.8</maven.compiler....
0
votes
1
answer
109
views
How to avoid NULL check in JOIN condition in Tableau
I am trying to change backend DB for Tableau dashboard. Tableau is generating JOIN SQLs with conditions such as:
ON a.col1 = b.col2 OR (a.col1 is null and b.col2 is null)
Is there a way we can avoid ...
-1
votes
2
answers
41
views
can I set the limit to avoid disk full
Is there any way that I can set the limit on historical node? getting error message that says "disk full" frequently, can I fix this?
Druid version I am using is 0.18.1
Thank you.
1
vote
3
answers
250
views
Can we modify timestamp in time picker in grafana?
I am using Druid as datasource for my grafana.
I want to ignore the first and last data points from the druid query result(like trimming the edges).
I am thinking of modifying the timestamp passed to ...
0
votes
1
answer
67
views
How to sort data based on one of the timestamp field in Druid Scan Query
I'm using Druid scan query with ordering param "ascending". It is returning data based on configured timestamp field called serverReceiveTime. I wanted to sort my data on one of the other ...
0
votes
1
answer
173
views
Clean up an apache druid cluster
Is there any way to clean up all the druid data (tasks, storage, etc.) for testing purposes?
Found the tutorial which demonstrates the segment deletion:
https://druid.apache.org/docs/latest/ingestion/...
0
votes
1
answer
118
views
How to run a Python code in Apache Druid as a UDF?
I am trying to run a Python UDF directly on Druid. Running the Python function directly on the machines has many advantages, not the least of which avoiding huge data transfers from and to the remote ...
0
votes
1
answer
90
views
Pagination issue with real time data in Druid Scan query
I have gone through following Druid Scan query documentation
https://druid.apache.org/docs/0.20.0/querying/scan-query.html . I didn't understand the part when it says. "note that if the ...
0
votes
1
answer
59
views
How to load data from Cassandra to Druid
I am newbie to Druid. I am able to ingest data to Druid from S3, Kafka. Now, I want to load data from Cassandra hosted over AWS private subnet.
Is it even possible? If yes, please share some resources....
1
vote
2
answers
126
views
How to configure druid batch indexing jobs dynamic EMR cluster for batch ingestion?
I am trying to automate druid batch ingestion using Airflow. My data pipeline creates EMR cluster on demand and shut it down once druid indexing is completed. But for druid we need to have Hadoop ...
0
votes
2
answers
226
views
Druid can not see/read GOOGLE_APPLICATION_CREDENTIALS defined on env path
I installed apache-druid-0.22.1 as a cluster (master, data and query nodes) and enabled “druid-google-extensions” by adding it to the array druid.extensions.loadList in common.runtime.properties.
...
0
votes
1
answer
186
views
Druid segments not available from some moment
Ingestion task was success but segments are unavailable from some moment.
I can't restart master node.. because it is not personal server.
Why segments are unavailable? or How can I fix it?
0
votes
1
answer
121
views
Delete data in druid datasource
I have an existing datasource in Druid. I am trying to delete some records by reindexing the data with filter and overwriting the existing data.
If the dataSource within ioConfig is my_datasource and ...
0
votes
1
answer
236
views
Druid segments not available
Hi Guys, There are ingestion tasks going on in my druid server setup on Kubernetes. Lot of segments in multiple datasources are not available, even though ingestion was successful. As a result I am ...
0
votes
1
answer
98
views
Is visualization with Druid and Superset only possible for time-series data?
When I connect Druid and Superset, can I use the data I have only for time-series charts? Or can I use it for other charts as well? (deck.gl Scatterplot, Mapbox etc.)
3
votes
3
answers
210
views
Can we able to do Transformation in Druid
I am having a scenario where I will be receiving data in csv files and there I need to generate some columns with the existing one.
Example:
Col_1 Col_2 Col_3 Col_4
abc 1 No 123
...
0
votes
4
answers
385
views
Druid -> Order data by another column than timestamp by native queries
I'm using scan query in druid.
I'm looking for a way to sort data using some query.
How I can do this?
Now I have:
DataSource: Data,
Intervals: "1000"/"2000",
Limit: 10
...