Questions tagged [azure-databricks]
For questions about the usage of Databricks Lakehouse Platform on Microsoft Azure
4,345
questions
0
votes
0
answers
4
views
Databricks - Create Table Options for ignoring first row
I am trying to create a table but the file I am loading just has system info in the first row with no delimiters and the columns don't correspond with the rest of the file. Is there an option to ...
0
votes
0
answers
13
views
What are the settings and syntax for creating a Databricks table referencing data that already exists on Azure ADLS?
I'm struggling with running a CREATE TABLE statement on Databricks that will point to a folder on Azure ADLS with data already in it.
Assuming this is the path and the account is already mounted: /mnt/...
0
votes
0
answers
35
views
Performance tuning tips for writing a PySpark dataframe to an Azure SQL Database table using Databricks
Problem statement:
Our data is stored on Azure Data Lake Storage Gen2, we need to process about 25 million records - 30 columns - on a daily basis into an Azure SQL Server database. We are using ...
1
vote
0
answers
29
views
shortening this block of code using indexing (Pyspark)
test_policies_rcc_1 = (
test_policies.join(
rates_table,
(
(test_policies.version_number == rates_table.version) &
(test_policies.pol_cover == ...
0
votes
1
answer
19
views
What is the best way to mnt azure storage account so that all notebooks inside azure databricks can access gen2 container
Run the following in your notebook to authenticate and create a mount point.
configs = {"fs.azure.account.auth.type": "OAuth",
"fs.azure.account.oauth.provider....
0
votes
0
answers
36
views
Query windows authenticated REST API from Azure DataBricks
We have a python notebook running on Azure Databricks which needs to query a REST API hosted on IIS on an on-prem windows server and requires windows authentication.
We've tried the below code from ...
0
votes
1
answer
47
views
Databricks SQL: Type Inference Challenges Using `COPY INTO`
I'm currently working on executing TPC-DS Benchmark queries in a Databricks SQL warehouse. The process involves generating data locally in CSV files, creating a Databricks database, and Delta tables ...
0
votes
1
answer
31
views
Passing multiple files in one pipeline adf
I'm new to ADF Pipeline.
Hello,
I have around n number of files in a nested folder structure as below.
landing/Datasource
File1.xml
File2.xml
... File..n.xml.
I have to pass each file as a parameter ...
0
votes
1
answer
27
views
SparkIllegalArgumentException: Regex group count is 0, but the specified group index is 1 error
I am running a query on databricks notebook with some complex regular expression. i am attaching the query below
%sql
SELECT * from (
SELECT
DISTINCT
md.asset_identity_serial ...
1
vote
0
answers
54
views
How to find documentation of dbruntime.dbutils.FileInfo class
I'm relatively new to Python, Spark and Cloud. I'm currently using the method dbutils.fs.ls(path), the databricks documentation doesn't specify that it returns a FileInfo object. When I do a "...
0
votes
1
answer
15
views
Delta Live Tables consuming different files from the same path are combining the schema
Summary
I am using Delta Live Tables to create a pipeline in Databricks and I am facing a problem of merging the schema of different files that are placed in the same folder in a datalake, even though ...
0
votes
1
answer
61
views
Databricks: Sharing information between workflow tasks using Scala
In Databricks, I would like to have a workflow with more than one task. I would like to pass information between those tasks. According to https://learn.microsoft.com/en-us/azure/databricks/workflows/...
0
votes
1
answer
24
views
How do I get an ADLS Gen2 URI for my blob?
I'm trying to create a Databricks external table that points to data sitting on ADLS, so I'm trying to figure out the proper URI syntax. Using this page, I've managed to get
'abfss://file_system@...
0
votes
1
answer
22
views
Adding a workspace file to Shared Databricks workspace
I'm trying to upload a python file as a workspace file to my Shared Databricks workspace. Using both the CLI and the REST API each Python file I upload turns into a notebook automatically, but I want ...
0
votes
1
answer
29
views
Is there a way to call GROUP BY as a variable within a calculated field in sql?
i am trying to sort by the year as my calculated field is not correct when trying to graph by date. i think if i get the date to be in chronological order then my problem will be solved but i'm not ...
0
votes
0
answers
26
views
databricks pyspark connectivity to MongoDB Atlas not working
i am using pyspark on databricks (cluster is unity enabled. ADB runtime=13.3) to connect to mongoDB Atlas.
i am using lib (latest version as per mongo document)=> org.mongodb.spark:mongo-spark-...
0
votes
1
answer
32
views
In pyspark, what is the difference between dlt.read_stream() and spark.readstream()?
I am tryting to understand In pyspark, what is the difference between dlt.read_stream() and spark.readstream()? can we replace dlt.read_stream() with spark.readstream() without affecting the data?
...
0
votes
1
answer
60
views
How to Change column datatype in databricks delta table
%sql
ALTER TABLE TableName alter COLUMN ColName int
Error:
ParseException:
[PARSE_SYNTAX_ERROR] Syntax error at or near 'int': extra input 'int'.(line 1, pos 68)
== SQL ==
ALTER TABLE TableName ...
0
votes
1
answer
59
views
Slow changing dimension | SCD type 1 deleting rows from prior date data
I have implemented SCD type 1 using merge into statement in azure databricks.
When I am doing first load, it’s loading the data that is expected with some just say 5000 rows for ingest date 28 Nov, ...
0
votes
1
answer
60
views
Can I use my ADF Filter activity to copy two or more file types?
I have a location where there are .csv, .txt and .hr files altogether.
I am using a filter in my ADF pipeline to filter .txt right now and copying it using Filter Settings condition @contains(item()....
1
vote
0
answers
78
views
Databricks parquet write jobs resulting in empty files
We have several Databricks PySpark jobs that write parquet files into the same container in Azure Blob Storage. Recently we sometimes get an output file with 0 bytes content even though the source ...
0
votes
0
answers
38
views
An error occurred while loading the model. (NotFound) Message: {"Errors":["Owner resource does not exist"]} Databricks model serving error
I am trying to understand what is this error message about: [ntkj5] An error occurred while loading the model. (NotFound) Message: {"Errors":["Owner resource does not exist"]}
In ...
0
votes
1
answer
50
views
connecting to azure sql from Databricks workspace
Created azure Databricks and mounted storage object using service principal. Now I am trying to ingest data from Azure sql and the connections is resulting in failure.
driver = "com.microsoft....
0
votes
1
answer
67
views
Can I Install R libraries in databricks without reinstalling each time the cluster is turned on?
I need to install R libraries (VIM and arrow), I have tried with the inits script and the global init script, also with the option to install libraries directly in the cluster but every time I turn on ...
0
votes
1
answer
76
views
Pyspark schema and dataframe interaction on optional fields
Good morning y'all, quick question about schemas and pyspark dataframes. Let's say I start with a schema and use that schema to load data into a dataframe from a file. So far, so good.
However, I ...
1
vote
1
answer
51
views
Autoloader - file notification and backfillInterval
Documentaion saying file event notification systems do not guarantee 100% delivery of all files and it recommneds to use backfills to guarantee that all files eventually get processed.
But its not ...
0
votes
1
answer
43
views
Is is possible to have the history metadata by row in Databricks
Is it possible to have the history metadata of a row in databricks?
Something like:
SELECT column1, column2, VERSION_NUMBER, VERSION_TIMESTAMP
FROM table
col1
col1
VERSION_NUMBER
VERSION_TIMESTAMP
...
1
vote
1
answer
61
views
Using shareplum to copy file to adls
I am trying to copy files (csv, image, ppt, docx, etc) from sharepoint into ADLS using shareplum, but I am having trouble on how to save the file.
import requests from shareplum
import Office365 from ...
-1
votes
1
answer
100
views
Cannot log in to Azure Databricks account console
I am trying to log in to the Azure Databricks account console for the first time.
Although I am logged in into Microsoft Entra ID (formerly Active Directory) as global administrator this does not work....
1
vote
2
answers
101
views
How can I pass parameters to databricks.yml in Databricks Asset Bundles?
Background:
I have a separate Databricks Workspace for each environment, and I am buidling an Azure DevOps pipeline to deploy a Databricks Asset Bundles to these environments.
Question
The asset ...
0
votes
1
answer
147
views
How to solve the following MaxRetryError: "TTPConnectionPool(host='localhost', port=48605): Max retries exceeded with url:"
I'm using the selenium library with python and I'm working on Databricks (Azure). I'm trying to connect to a web page to take values and I can do it the first time. From the second time, the following ...
0
votes
0
answers
22
views
Spark Executors were added 2 minutes after the final step was completed [closed]
I am running a spark job which got completed using default 2 executors but after 2 minutes I found that 4 more executors were added after the final job got completed. .
Event log contains auto scaling ...
1
vote
1
answer
65
views
Spark- Spill disk and Spill memory problem
I am new to spark and working on a logic to join 13 files and write the final file into a blob storage. Input files are in CSV format and output is written as parquet. Out of the 13 files, file1 is ...
0
votes
2
answers
155
views
Merge 2 dataframes in Pyspark
In Pyspark, I have 2 dataframe. 1st dataframe say df1 is empty dataframe created from a schema. 2nd dataframe df2 is non-empty dataframe filled from a csv file.
Now I want to merge such that all below ...
0
votes
1
answer
133
views
+100
Memory not released in Azure DataBricks
In Azure DataBricks (12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12) I am running the following code in the Notebook on compute Standard_DS5_v2:
df_spark = spark.sql(my_qyery_to_delta_table)
...
0
votes
1
answer
60
views
How to determine which Azure Keyvault contains certain keys with Databricks dbutils
Is it possible to determine which keyvault holds certain keys with Databricks DBUtils.
For example, if I run dbutils.secrets.list("my_scope_dev") I will be able to see existing keys. Is ...
0
votes
2
answers
63
views
Read Delta tables from ADLS to PowerBI
I am integrating ADLS with Databricks and want to load delta tables stored in ADLS to PowerBI.
Each time I update the table a new parquet file is created in the ADLS directory location in addition to ...
0
votes
1
answer
68
views
Databricks JDBC with Pyspark returning rows all rows as columns name
I am trying to access and query from Azure DataBricks but I have been getting all rows as column name.
I am following method for code.
Please give solution if anyone is aware of this.
df = spark.read....
0
votes
1
answer
78
views
Install Maven Library on (ADF) Databricks Job Cluster
I've tried using /databricks/spark/bin/spark-shell --packages com.crealytics:spark-excel_2.13:3.4.1_0.19.0 in my init script, however I get the error Error: Could not find or load main class org....
0
votes
1
answer
83
views
Change location of existing Tables in Unity Catalog
I want to know how can we change location of existing (Managed/External) Tables in Unity Catalog from Storage Account/Blob to Another.
We want to have a functionality to make it flexible to store data ...
0
votes
1
answer
89
views
Databricks dbutils.fs.cp
I am using dbutils.fs.cp(sourcefilepath,destinationfilepath) query to copy a file from one location to another from the azure storages.My source file path is /mnt/testing/StudentA/Test.csv and ...
0
votes
1
answer
34
views
Unable to understand how pyspark statements work without importing on my cluster
I am using Azure Databricks connected to a Databricks runtime 13.0. Then, I could run this command without importing pyspark:
from pyspark.sql import functions as F
How is this possible?
I was ...
1
vote
1
answer
63
views
How to connect and read/write to Azure ADLS Gen2 or Blob Store using Shared Access Signature with Databricks
I created SAS keys for my Azure Storage account, see image below.
Can someone show me how to connect to the Storage account with Databricks using the one of the Keys listed below:
I tried the ...
0
votes
1
answer
104
views
what is the correct way to access a workspace file in databricks
My databricks runtime version is 10.4 LTS. I am trying to access a workspace file using open() method from python. I tried with multiple different ways, but they all failed.
Suppose my workspace file ...
0
votes
1
answer
83
views
Databricks Error: AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `mytable` cannot be found
When I attempt a select * from a VIEW in databricks I get the error:
AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `mytable` cannot be found. Verify the spelling and correctness of ...
0
votes
1
answer
70
views
Assign a custom label value to the SQL dropdown widget in Databricks
I developed an SQL dropdown widget using guidance from the Databricks documentation page. Currently, the dropdown label is configured as "state" in the following query. However, my intention ...
0
votes
1
answer
76
views
Databricks Delta Table from Datalake with Partitioned Folders
I have data on Azure Storage Account in a Container called Staging.
Using the SourceSystem/Dataset hierarchy I also have date based folder per month.
Inside the folder the data is stored in a delta ...
0
votes
1
answer
192
views
Cannot create secret scope: Azure KeyVault is not available. Terraform
I'm deploying a keyvault and a databricks resource with terraform. The resources are deploy via a Service Principal that will give contributor rights to every thing it deploys.
I also added to myself ...
0
votes
1
answer
54
views
How to output the 'Underlying SQLException' from Azure Databricks instead of the generic exception message?
We're calling Azure Databricks notebooks from an Data Factory pipeline that performs ingestion into Azure Synapse. But whenever a notebook run fails, it just shows the below error message:
com....
0
votes
2
answers
91
views
Get the previous not null value if the current value is null - Azure databricks
I have a table which has the below values
ID Date FinalDate
1 20120101 NULL
1 20120201 20120201
1 20120301 NULL
1 20120401 20120401
1 20120501 NULL
1 ...