Questions tagged [delta-lake]

Delta Lake is an open source project that supports ACID on top of Apache Spark. It provides ACID transactions, scalable metadata handling, time travel, unified batch and streaming source and sink, and is fully compatible with Apache Spark APIs.

Filter by
Sorted by
Tagged with
0 votes
1 answer
73 views

How to set "zstd" compression level in AWS Glue job?

Background "zstd" compression codec has 22 compression levels. I read this Uber blog. Regarding compressing time and file size, I verified using df.to_parquet with our data and got same ...
Hongbo Miao's user avatar
  • 45.6k
1 vote
1 answer
62 views

How to write to Delta Lake using "zstd" compression codec in AWS Glue?

I have AWS Glue job. It is using "Glue 4.0 - Spark 3.3, Scala 2, Python 3" version. It reads individual parquet files and write to Delta Lake. I am using "write.parquet.compression-...
Hongbo Miao's user avatar
  • 45.6k
0 votes
0 answers
33 views

Persist newly calculated value in delta lake - Databricks operation MERGE fails

Situation description I'm working with Databricks on Azure and perform an MERGE operation. I'm loading the initial delta table dt with spark.read.format("delta").option('MergeSchema',True)....
mat.tho's user avatar
  • 194
1 vote
2 answers
26 views

How to set partition size for date field in Databricks

I have timestamp field source_timestamp which is used a lot in the queries to the delta table. I would like to partition by this column with size of the partition = 1 month. How to achieve that? If I ...
archjkeee's user avatar
0 votes
0 answers
18 views

Databricks Low Shuffle Merge design

Does anyone know how Low shuffle merge has been implemented in Databricks runtime? I tried searching for any white paper on it but the only available information is very high level and couldn't find ...
singingwhale's user avatar
0 votes
1 answer
36 views

Create a delta table in unity catalog using PySpark

I have a PySpark DataFrame and I want to create it as Delta Table on my unity catalog. I have the following strucutre: prd |—- landing |—- bronze |—- silver |—- gold |—- qa I have my prd catalog with ...
OdiumPura's user avatar
  • 448
1 vote
1 answer
18 views

Delta Lake 'OPTIMIZE' command does not use all available nodes

Setup AWS EMR 6.10 100 x r5d.4xlarge nodes Spark 3.3.1 Delta Lake 2.2.0 Spark submit conf args (not all, just related to performance) --num-executors 100 --executor-memory 64g --conf spark....
Alexander Pavlov's user avatar
0 votes
1 answer
36 views

Deduplicate records from existing delta table in memory

We have a delta table on production that has a lot of duplicate rows, we could read and do dropDuplicates() on the delta table but the problem is the volume of the data is pretty huge we can't read it ...
Spook's user avatar
  • 1
0 votes
1 answer
40 views

How to clone a delta table properly? SQL Editor in DataBricks says a column of delta table does not exist but it is

I need to clone a delta table from a productive environment to test environment in order to make an analysis. I've tried to use the clonate function but I didn't figured out how to clonate a table ...
Xkid's user avatar
  • 362
0 votes
0 answers
16 views

Retraction in aggregation with Databrick Delta

I have been working with kafka streams for a while, and been quite happy with it. Recently I deiced to explore the competition of the same "Kind", namely Flink and DataBrick Delta. I do not ...
MaatDeamon's user avatar
  • 9,532
0 votes
0 answers
20 views

Debugging data-loss issue in Delta Table (difference observed through operation metrics as well)

In our service we noticed the data loss in Delta Table where the total no of rows are not aligned with the input that we had fed during the merge. I confirmed the difference through counting the rows ...
Deepak Singhal's user avatar
0 votes
1 answer
46 views

In DeltaTables, why does append mode with mergeSchema create a full copy of the data in storage?

I'm working with synapse notebooks and pyspark and I'm trying to support schema evolution in an efficient manner. The format doesn't have to necessarily be DeltaTables but it seems like a natural ...
Bistabil's user avatar
  • 184
1 vote
1 answer
56 views

Update merge in pyspark takes very long

Why does the MERGE INTO statement take significantly longer to update records in the old DataFrame when the old DataFrame has over 200 columns compared to when the old DataFrame has only 5 columns? ...
JoostBerkers's user avatar
0 votes
0 answers
52 views

Delta lake with Glue Catalog as metastore

We are using Spark (v3.3.0) SQL to consume data from a delta lake (v2.2.0), running on top of S3. The Spark jobs are run using Glue jobs and also EMR cluster. We want our code to refer to the delta ...
Subodh Shetty's user avatar
0 votes
1 answer
39 views

Spark Delta Merge runs too slow in Test

We have a Scalatest + Spark based Test. The Data transformation is fairly complex with lots of transformation and the final step is Delta merge. The tests have minimal input data (around 20 KB) and ...
Ganesha's user avatar
  • 145
0 votes
1 answer
30 views

Problems reading delta lake files in Databricks

In another issue here I was having difficulty reading all the files, someone managed to help me. I managed to read all the files, but can you guys give me some more help? I'm reading 12 different ...
Joyci Santos's user avatar
0 votes
0 answers
26 views

PySpark - Try to use Delta Lake in AWS Glue with saveAsTable

I am trying to have an experience with Delta data in AWS Glue. I am creating an AWS Glue job using Glue 4.0 version. In Job details I added two job parameters: --conf Value:spark.sql.extensions=io....
bigdataadd's user avatar
0 votes
0 answers
62 views

AutoLoader - data missed by readStream during files are written to Azure Data Lake Storage

I'm using Databricks AutoLoader to ingest data from files (parquets) to Delta Tables. https://docs.databricks.com/en/ingestion/auto-loader/index.html In fact our source system writes files to Azaure ...
Marcin U's user avatar
0 votes
1 answer
27 views

problem reading files in delta lake house - data streaming

I'm building my first delta lake house in databricks, alone. I need to read files that are in csv format in a bucket in AWS, I can view the folder with my files by doing display(dbutils.fs.ls. However,...
Joyci Santos's user avatar
1 vote
2 answers
72 views

Optimizing Delta Lake Partitioning for Timestamp Range Queries

I'm working with a Delta table in Databricks that is partitioned by a Year column, and I have a timestamp column in the table. I want to optimize my queries to fetch data within a specific timestamp ...
elyptikus's user avatar
  • 968
0 votes
1 answer
54 views

Archiving/Purging in Delta Tables in Azure Synapse (without using Databricks)

I am working on creating a flexible and parameterized framework/pipeline that Archives/Purges the data using Azure Synapse. I have a few Delta tables in the current setup that cannot be handled like ...
parul jain's user avatar
0 votes
1 answer
59 views

Write Delta format to Data Lake in AWS S3

I've been stuck in this for hours, hope someone can help. My goal is to upload data in delta format to a AWS S3 Data Lake. I have to mention that I've achieved to upload files (parquet) to this ...
NaejF's user avatar
  • 91
0 votes
1 answer
31 views

Which combination of pyspark and delta versions should I use?

I'm using the jupyter/pyspark-notebook docker image to develop a spark script. My dockerfile looks like this: FROM jupyter/pyspark-notebook USER root COPY requirements.txt . RUN pip install --no-...
Boris's user avatar
  • 736
0 votes
0 answers
20 views

Delete specific Delta Lake table versions?

Noting that time travel in Delta is not meant to be a permanent history, is there a method to delete specific versions of a table to keep fewer long term history versions for time travel? The idea ...
kermatt's user avatar
  • 1,585
0 votes
1 answer
60 views

Joining two tables for delete using DeltaTable

I have an use case, where I have a file path which has "company_id" and "date". Have another s3 file where I need to delete data which is below year 2010 based on "company_id&...
Shasu's user avatar
  • 462
1 vote
0 answers
33 views

Delta lake on z order optimize creating very large file

I have a question regarding the maximum file size created when using the Z-Order optimization command. I have a partitioned Delta Lake table where I structure streaming data and use the merge command ...
sangeeth sasidharan's user avatar
0 votes
0 answers
50 views

Delta Table saveAsTable giving null pointer exception

I am getting the below error stack trace while creating a Delta table. Spark version 3.2.1 Traceback (most recent call last): File "/tmp/spark-08dced9c-acdb-419d-8952-7d04e5531238/...
Susmit Sarkar's user avatar
2 votes
1 answer
51 views

Any reason to set optimizeWrite = 'true', when autoCompact = 'auto' and no partitions are used

I work with time series data and Ingestion-Time Clustering (no partitioning) has proven to work well. In the Databricks docs is written "Optimized writes are most effective for partitioned tables,...
Oliver Angelil's user avatar
0 votes
1 answer
120 views

AWS Glue: Cannot find catalog plugin class for catalog 'spark_catalog': org.apache.spark.sql.delta.catalog.DeltaCatalog

I have a Glue job in Python, which write individual parquet files to Delta Lake in S3. import sys from awsglue.context import GlueContext from awsglue.job import Job from awsglue.utils import ...
Hongbo Miao's user avatar
  • 45.6k
0 votes
1 answer
62 views

Writing to UserMetadata Field During Table Creation

When I run the following code in my DataBricks environment, the initial save does not write "Initial Commit" to the userMetaData field; however, the two following "append" ...
Shawn Eary's user avatar
0 votes
2 answers
32 views

AWS Glue locally: convert pandas df to delta

I run my AWS Glue jobs locally in a docker container (AWS Glue lib 4.0) and want to convert/write a pandas dataframe to delta format. I added spark = SparkSession.builder \ .appName("...
Finja's user avatar
  • 1
0 votes
1 answer
58 views

Which time-based partitioning strategy to use for Delta or Iceberg

I am using spark-streaming to ingest live event streams every 5 minutes and append into delta or apache iceberg table. this table can be ingested and processed by downstream data pipeline or it can be ...
nir's user avatar
  • 3,753
0 votes
1 answer
55 views

Delta Tables are been deleted but delta files are available at back end silver layer

Delta Tables are been deleted but delta files are available at back end silver layer Method used is : We are doing ingestion by loading parquet file from raw to silver, Deleting the delta folder ...
Sai Kiran's user avatar
0 votes
1 answer
50 views

Is CDF feature possible using delta-spark on Cloudera distribution?

We have our application using the on-premise CDP (Cloudera) cluster for submitting pyspark jobs. Version of spark is 2.x We are now exploring the option to have CDC datasets processed and merge with ...
lbvirgo's user avatar
  • 354
1 vote
1 answer
32 views

Is there a way to access a delta table old column name?

I have a table with customer IDs, creditcard_number and creditcard_type (amex, mastercard, etc.) I am trying to find a way to: Modify the column name creditcard_type to type and creditcard_number to ...
Roland R.'s user avatar
0 votes
1 answer
121 views

Performance issue while merging data to a delta lake

The solution that I'm working on involves a delta lake table that had a full load in the beginning of around 110gb and now there's incremental load daily of around 1-2mb. The first time I loaded the ...
Indraneel Dave's user avatar
0 votes
1 answer
52 views

ImportError: cannot import name 'CapturedException' from 'pyspark.sql.utils'

dependencies: pyspark==3.4.0 delta-spark==2.0.0 python==3.11.4 Code snippet to reproduce: from delta import configure_spark_with_delta_pip Error Log: ImportError ...
stevezkw's user avatar
  • 466
-1 votes
0 answers
17 views

Do Delta Lakes support updating rows?

I want to store data in delta tables. I want to be able to perform CRUD operations on the data from time to time. I may want to edit rows, add rows and delete rows. Is this possible in delta lakes?.
Jay2454643's user avatar
0 votes
1 answer
52 views

I need to create a view on top of a specific version of a Delta table. Is there a way to do it within Synapse, and using T-SQL?

I have a Delta table stored in my Azure Data Lake Storage Gen2 (ADLS Gen2) account. When this Delta table is updated by a Synapse pipeline, I want to automatically create a view on top of the latest ...
Anish's user avatar
  • 21
0 votes
0 answers
16 views

Can we create a delta table linked to multiple buckets based on partitions?

Let's say I want to create a delta table giving a particular location(a gcs bucket) , but I want to load create partitions in a different gcs bucket , lets say table is partitioned on country ,now use ...
Anurag Pandey's user avatar
1 vote
0 answers
90 views

skipChangeCommits in Delta live table

I am trying to create a DLT pipeline to implement SCD type 1. However, after my initial full refresh, when I am trying to do an incremental refresh, it fails giving below error: "An error ...
Anil Panda's user avatar
0 votes
0 answers
18 views

how to build delta-rs python for hdfs support

i would build delta-rs v0.10.1 with HDFS support. for this purpose i have costomized Cargo.toml file and add hdfs feature then building and installing it in environment using make install command. ...
Mohammad Esmaeili's user avatar
0 votes
1 answer
56 views

How to extract value from a spark dataframe and add it to a second one as a column?

I have 2 large spark dataframes df1 and df2. df1 has a column with a colName name that has only one distinct value. I need to add this column to the df2. I'm wondering what would be the most efficient ...
Oliwier Mroczkowski's user avatar
-4 votes
1 answer
52 views

How to write SQL query to read most recent timestamps (using window function)? [closed]

I have a table in my database that has rows inserted without updating old ones. That leads to having records with same ID but diffrent timestamps. How to write SQL query that uses window function to ...
Oliwier Mroczkowski's user avatar
0 votes
0 answers
24 views

How to partition a table with different ingestion sources and volumes?

I have a database implemented with Databricks Delta Lake and Azure Storage. I need to create a table that would have a new data ingested on daily basis. The data would come from diffrent sources. Each ...
Oliwier Mroczkowski's user avatar
2 votes
1 answer
51 views

How to clone a SQL database in Azure Synapse - no menu option, despite being Synapse Administrator

I'm trying to clone a Synapse Serverless SQL database, so I can test with it. When I click a SQL database in Synapse Studio, there's no option to clone. No 'Clone' option in menu I am the Synapse ...
Pingyao's user avatar
  • 25
2 votes
1 answer
73 views

What is the difference between a data lakehouse and a delta lake?

I am new to Databricks. I am reading Microsoft documentation on data lakehouse. In the documentation they make reference to delta lake without explaining what the difference is or even if there is any....
Jay2454643's user avatar
0 votes
0 answers
25 views

Performance optimization of a Pyspark code which uses hive external table query multiple times

I am having a Pyspark code which applies complex transformations. In this code, we are using one particular hive external table multiple times,to be precise the subset data using partitioned column ...
Ananth's user avatar
  • 41
1 vote
1 answer
59 views

Unreasonable memory consumption for delta lake on pandas

I am working on Databricks using Delta Lake. I have a dataset that is approx 1.9GB in size (parquet format). I am trying to convert this dataset to a Delta Table and I was able to successfully create ...
Mattia Surricchio's user avatar
1 vote
1 answer
59 views

How to know number of rows affected by CDF merge in pyspark?

I have a CDF logic where I need to know of row impacted due to merge i.e I need to know number of inserted, Updated and deleted row in order to make some decision. i am able to get to the required ...
Aditya Raj's user avatar

1
2 3 4 5
26