Questions tagged [azure-databricks]

For questions about the usage of Databricks Lakehouse Platform on Microsoft Azure

Filter by
Sorted by
Tagged with
0 votes
0 answers
4 views

Databricks - Create Table Options for ignoring first row

I am trying to create a table but the file I am loading just has system info in the first row with no delimiters and the columns don't correspond with the rest of the file. Is there an option to ...
Caz's user avatar
  • 57
0 votes
0 answers
13 views

What are the settings and syntax for creating a Databricks table referencing data that already exists on Azure ADLS?

I'm struggling with running a CREATE TABLE statement on Databricks that will point to a folder on Azure ADLS with data already in it. Assuming this is the path and the account is already mounted: /mnt/...
simplycoding's user avatar
  • 2,817
0 votes
0 answers
35 views

Performance tuning tips for writing a PySpark dataframe to an Azure SQL Database table using Databricks

Problem statement: Our data is stored on Azure Data Lake Storage Gen2, we need to process about 25 million records - 30 columns - on a daily basis into an Azure SQL Server database. We are using ...
wbaeckelmans's user avatar
1 vote
0 answers
29 views

shortening this block of code using indexing (Pyspark)

test_policies_rcc_1 = ( test_policies.join( rates_table, ( (test_policies.version_number == rates_table.version) & (test_policies.pol_cover == ...
555codewiz's user avatar
0 votes
1 answer
19 views

What is the best way to mnt azure storage account so that all notebooks inside azure databricks can access gen2 container

Run the following in your notebook to authenticate and create a mount point. configs = {"fs.azure.account.auth.type": "OAuth", "fs.azure.account.oauth.provider....
Surender Raja's user avatar
0 votes
0 answers
36 views

Query windows authenticated REST API from Azure DataBricks

We have a python notebook running on Azure Databricks which needs to query a REST API hosted on IIS on an on-prem windows server and requires windows authentication. We've tried the below code from ...
siddharth's user avatar
  • 670
0 votes
1 answer
47 views

Databricks SQL: Type Inference Challenges Using `COPY INTO`

I'm currently working on executing TPC-DS Benchmark queries in a Databricks SQL warehouse. The process involves generating data locally in CSV files, creating a Databricks database, and Delta tables ...
manucorujo's user avatar
0 votes
1 answer
31 views

Passing multiple files in one pipeline adf

I'm new to ADF Pipeline. Hello, I have around n number of files in a nested folder structure as below. landing/Datasource File1.xml File2.xml ... File..n.xml. I have to pass each file as a parameter ...
wanderingdev's user avatar
0 votes
1 answer
27 views

SparkIllegalArgumentException: Regex group count is 0, but the specified group index is 1 error

I am running a query on databricks notebook with some complex regular expression. i am attaching the query below %sql SELECT * from ( SELECT DISTINCT md.asset_identity_serial ...
sayan nandi's user avatar
1 vote
0 answers
54 views

How to find documentation of dbruntime.dbutils.FileInfo class

I'm relatively new to Python, Spark and Cloud. I'm currently using the method dbutils.fs.ls(path), the databricks documentation doesn't specify that it returns a FileInfo object. When I do a "...
agusteando's user avatar
0 votes
1 answer
15 views

Delta Live Tables consuming different files from the same path are combining the schema

Summary I am using Delta Live Tables to create a pipeline in Databricks and I am facing a problem of merging the schema of different files that are placed in the same folder in a datalake, even though ...
Arnold Souza's user avatar
0 votes
1 answer
61 views

Databricks: Sharing information between workflow tasks using Scala

In Databricks, I would like to have a workflow with more than one task. I would like to pass information between those tasks. According to https://learn.microsoft.com/en-us/azure/databricks/workflows/...
stuffed's user avatar
0 votes
1 answer
24 views

How do I get an ADLS Gen2 URI for my blob?

I'm trying to create a Databricks external table that points to data sitting on ADLS, so I'm trying to figure out the proper URI syntax. Using this page, I've managed to get 'abfss://file_system@...
simplycoding's user avatar
  • 2,817
0 votes
1 answer
22 views

Adding a workspace file to Shared Databricks workspace

I'm trying to upload a python file as a workspace file to my Shared Databricks workspace. Using both the CLI and the REST API each Python file I upload turns into a notebook automatically, but I want ...
Frank's user avatar
  • 23
0 votes
1 answer
29 views

Is there a way to call GROUP BY as a variable within a calculated field in sql?

i am trying to sort by the year as my calculated field is not correct when trying to graph by date. i think if i get the date to be in chronological order then my problem will be solved but i'm not ...
fi123's user avatar
  • 1
0 votes
0 answers
26 views

databricks pyspark connectivity to MongoDB Atlas not working

i am using pyspark on databricks (cluster is unity enabled. ADB runtime=13.3) to connect to mongoDB Atlas. i am using lib (latest version as per mongo document)=> org.mongodb.spark:mongo-spark-...
Rakesh Prasad's user avatar
0 votes
1 answer
32 views

In pyspark, what is the difference between dlt.read_stream() and spark.readstream()?

I am tryting to understand In pyspark, what is the difference between dlt.read_stream() and spark.readstream()? can we replace dlt.read_stream() with spark.readstream() without affecting the data? ...
Aditya Raj's user avatar
0 votes
1 answer
60 views

How to Change column datatype in databricks delta table

%sql ALTER TABLE TableName alter COLUMN ColName int Error: ParseException: [PARSE_SYNTAX_ERROR] Syntax error at or near 'int': extra input 'int'.(line 1, pos 68) == SQL == ALTER TABLE TableName ...
Antony's user avatar
  • 966
0 votes
1 answer
59 views

Slow changing dimension | SCD type 1 deleting rows from prior date data

I have implemented SCD type 1 using merge into statement in azure databricks. When I am doing first load, it’s loading the data that is expected with some just say 5000 rows for ingest date 28 Nov, ...
Saurabh's user avatar
  • 107
0 votes
1 answer
60 views

Can I use my ADF Filter activity to copy two or more file types?

I have a location where there are .csv, .txt and .hr files altogether. I am using a filter in my ADF pipeline to filter .txt right now and copying it using Filter Settings condition @contains(item()....
Crime_Master_GoGo's user avatar
1 vote
0 answers
78 views

Databricks parquet write jobs resulting in empty files

We have several Databricks PySpark jobs that write parquet files into the same container in Azure Blob Storage. Recently we sometimes get an output file with 0 bytes content even though the source ...
Veronika Vrana's user avatar
0 votes
0 answers
38 views

An error occurred while loading the model. (NotFound) Message: {"Errors":["Owner resource does not exist"]} Databricks model serving error

I am trying to understand what is this error message about: [ntkj5] An error occurred while loading the model. (NotFound) Message: {"Errors":["Owner resource does not exist"]} In ...
Samlex's user avatar
  • 171
0 votes
1 answer
50 views

connecting to azure sql from Databricks workspace

Created azure Databricks and mounted storage object using service principal. Now I am trying to ingest data from Azure sql and the connections is resulting in failure. driver = "com.microsoft....
FalloutATS21's user avatar
0 votes
1 answer
67 views

Can I Install R libraries in databricks without reinstalling each time the cluster is turned on?

I need to install R libraries (VIM and arrow), I have tried with the inits script and the global init script, also with the option to install libraries directly in the cluster but every time I turn on ...
jean's user avatar
  • 21
0 votes
1 answer
76 views

Pyspark schema and dataframe interaction on optional fields

Good morning y'all, quick question about schemas and pyspark dataframes. Let's say I start with a schema and use that schema to load data into a dataframe from a file. So far, so good. However, I ...
Niehm's user avatar
  • 143
1 vote
1 answer
51 views

Autoloader - file notification and backfillInterval

Documentaion saying file event notification systems do not guarantee 100% delivery of all files and it recommneds to use backfills to guarantee that all files eventually get processed. But its not ...
Greencolor's user avatar
0 votes
1 answer
43 views

Is is possible to have the history metadata by row in Databricks

Is it possible to have the history metadata of a row in databricks? Something like: SELECT column1, column2, VERSION_NUMBER, VERSION_TIMESTAMP FROM table col1 col1 VERSION_NUMBER VERSION_TIMESTAMP ...
Bradford1138's user avatar
1 vote
1 answer
61 views

Using shareplum to copy file to adls

I am trying to copy files (csv, image, ppt, docx, etc) from sharepoint into ADLS using shareplum, but I am having trouble on how to save the file. import requests from shareplum import Office365 from ...
J. Doe's user avatar
  • 189
-1 votes
1 answer
100 views

Cannot log in to Azure Databricks account console

I am trying to log in to the Azure Databricks account console for the first time. Although I am logged in into Microsoft Entra ID (formerly Active Directory) as global administrator this does not work....
38fuenf's user avatar
1 vote
2 answers
101 views

How can I pass parameters to databricks.yml in Databricks Asset Bundles?

Background: I have a separate Databricks Workspace for each environment, and I am buidling an Azure DevOps pipeline to deploy a Databricks Asset Bundles to these environments. Question The asset ...
Kim Jensen's user avatar
0 votes
1 answer
147 views

How to solve the following MaxRetryError: "TTPConnectionPool(host='localhost', port=48605): Max retries exceeded with url:"

I'm using the selenium library with python and I'm working on Databricks (Azure). I'm trying to connect to a web page to take values and I can do it the first time. From the second time, the following ...
Sebastian Rivera Muñoz's user avatar
0 votes
0 answers
22 views

Spark Executors were added 2 minutes after the final step was completed [closed]

I am running a spark job which got completed using default 2 executors but after 2 minutes I found that 4 more executors were added after the final job got completed. . Event log contains auto scaling ...
Ash's user avatar
  • 55
1 vote
1 answer
65 views

Spark- Spill disk and Spill memory problem

I am new to spark and working on a logic to join 13 files and write the final file into a blob storage. Input files are in CSV format and output is written as parquet. Out of the 13 files, file1 is ...
Ash's user avatar
  • 55
0 votes
2 answers
155 views

Merge 2 dataframes in Pyspark

In Pyspark, I have 2 dataframe. 1st dataframe say df1 is empty dataframe created from a schema. 2nd dataframe df2 is non-empty dataframe filled from a csv file. Now I want to merge such that all below ...
GauravK's user avatar
0 votes
1 answer
133 views
+100

Memory not released in Azure DataBricks

In Azure DataBricks (12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12) I am running the following code in the Notebook on compute Standard_DS5_v2: df_spark = spark.sql(my_qyery_to_delta_table) ...
RunTheGauntlet's user avatar
0 votes
1 answer
60 views

How to determine which Azure Keyvault contains certain keys with Databricks dbutils

Is it possible to determine which keyvault holds certain keys with Databricks DBUtils. For example, if I run dbutils.secrets.list("my_scope_dev") I will be able to see existing keys. Is ...
Patterson's user avatar
  • 2,105
0 votes
2 answers
63 views

Read Delta tables from ADLS to PowerBI

I am integrating ADLS with Databricks and want to load delta tables stored in ADLS to PowerBI. Each time I update the table a new parquet file is created in the ADLS directory location in addition to ...
Thoughtful_Giraffe's user avatar
0 votes
1 answer
68 views

Databricks JDBC with Pyspark returning rows all rows as columns name

I am trying to access and query from Azure DataBricks but I have been getting all rows as column name. I am following method for code. Please give solution if anyone is aware of this. df = spark.read....
Usman_Ahmed's user avatar
0 votes
1 answer
78 views

Install Maven Library on (ADF) Databricks Job Cluster

I've tried using /databricks/spark/bin/spark-shell --packages com.crealytics:spark-excel_2.13:3.4.1_0.19.0 in my init script, however I get the error Error: Could not find or load main class org....
xuxu's user avatar
  • 420
0 votes
1 answer
83 views

Change location of existing Tables in Unity Catalog

I want to know how can we change location of existing (Managed/External) Tables in Unity Catalog from Storage Account/Blob to Another. We want to have a functionality to make it flexible to store data ...
Saurabh Mehta's user avatar
0 votes
1 answer
89 views

Databricks dbutils.fs.cp

I am using dbutils.fs.cp(sourcefilepath,destinationfilepath) query to copy a file from one location to another from the azure storages.My source file path is /mnt/testing/StudentA/Test.csv and ...
Deepak Kumar's user avatar
0 votes
1 answer
34 views

Unable to understand how pyspark statements work without importing on my cluster

I am using Azure Databricks connected to a Databricks runtime 13.0. Then, I could run this command without importing pyspark: from pyspark.sql import functions as F How is this possible? I was ...
Veeru's user avatar
  • 1
1 vote
1 answer
63 views

How to connect and read/write to Azure ADLS Gen2 or Blob Store using Shared Access Signature with Databricks

I created SAS keys for my Azure Storage account, see image below. Can someone show me how to connect to the Storage account with Databricks using the one of the Keys listed below: I tried the ...
Patterson's user avatar
  • 2,105
0 votes
1 answer
104 views

what is the correct way to access a workspace file in databricks

My databricks runtime version is 10.4 LTS. I am trying to access a workspace file using open() method from python. I tried with multiple different ways, but they all failed. Suppose my workspace file ...
jdhao's user avatar
  • 24.7k
0 votes
1 answer
83 views

Databricks Error: AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `mytable` cannot be found

When I attempt a select * from a VIEW in databricks I get the error: AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `mytable` cannot be found. Verify the spelling and correctness of ...
Patterson's user avatar
  • 2,105
0 votes
1 answer
70 views

Assign a custom label value to the SQL dropdown widget in Databricks

I developed an SQL dropdown widget using guidance from the Databricks documentation page. Currently, the dropdown label is configured as "state" in the following query. However, my intention ...
Arulkumar's user avatar
  • 13k
0 votes
1 answer
76 views

Databricks Delta Table from Datalake with Partitioned Folders

I have data on Azure Storage Account in a Container called Staging. Using the SourceSystem/Dataset hierarchy I also have date based folder per month. Inside the folder the data is stored in a delta ...
LordRofticus's user avatar
0 votes
1 answer
192 views

Cannot create secret scope: Azure KeyVault is not available. Terraform

I'm deploying a keyvault and a databricks resource with terraform. The resources are deploy via a Service Principal that will give contributor rights to every thing it deploys. I also added to myself ...
BeGreen's user avatar
  • 785
0 votes
1 answer
54 views

How to output the 'Underlying SQLException' from Azure Databricks instead of the generic exception message?

We're calling Azure Databricks notebooks from an Data Factory pipeline that performs ingestion into Azure Synapse. But whenever a notebook run fails, it just shows the below error message: com....
LearneR's user avatar
  • 2,355
0 votes
2 answers
91 views

Get the previous not null value if the current value is null - Azure databricks

I have a table which has the below values ID Date FinalDate 1 20120101 NULL 1 20120201 20120201 1 20120301 NULL 1 20120401 20120401 1 20120501 NULL 1 ...
John Bryan's user avatar

1
2 3 4 5
87