Questions tagged [databricks]
For questions about the Databricks Unified Analytics Platform
2,665
questions
1
vote
1answer
8 views
Calling R workbook in Databricks
Let's say I create a basic function in R:
Addn <- function(X,n)
{
X + n
}
and this is saved to a Databricks workbook; some filepath: "/shared/x/y/z/Addnfunction"
In RStudio, I ...
4
votes
1answer
37 views
azure pyspark register udf from jar Failed UDFRegistration
I'm having trouble registering some udfs that are in a java file. I've a couple approaches but they all return :
Failed to execute user defined function(UDFRegistration$$Lambda$6068/1550981127: (...
0
votes
0answers
30 views
Netezza Stored Procedure migration to Databricks
We have an use case where one of our client has started migration from Netezza to Databricks(metadata backed by Glue metastore). The problem we are facing is we have 500+ Netezza stored procedures and ...
0
votes
1answer
48 views
Efficiently aggregrate (fitler/select) a large dataframe in a loop and create new dataframe
I have 1 large dataframe that is created by importing a csv file (sparkscv). This dataframe has many rows of daily data. The data is identified by date, region, service_offered and count.
I filter ...
1
vote
1answer
36 views
Running the same Databricks Python Notebook concurrently
I am running the same notebook three times in parallel using the code below:
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
def notebook1_function(country, days):
...
0
votes
1answer
25 views
Write rows to SQL server
I have a table in SQL server. I am trying to append a spark dataframe from databricks to the SQL table. I am able to append the new rows but the problem is that it appends randomly to the SQL table. I ...
0
votes
2answers
155 views
Databricks: calling widget value in query
I've created a widget called yrmo with the values such as 202001.
I need to call this value from a query but it doesn't recognize it, I think because the field I am applying it to is Int but I can ...
1
vote
2answers
59 views
Read Delta table from multiple folders
I'm working on a Databricks. I'm reading my delta table like this:
path = "/root/data/foo/year=2021/"
df = spark.read.format("delta").load(path)
However within the year=2021 ...
0
votes
0answers
19 views
Delta Lake MERGE / UPDATE rewriting data even when condition is not met
I'm trying to reduce unnecessary writing of data and only write to the delta lake under a specific condition. Why do these statements always rewrite the data?
%sql
MERGE INTO tblTest as target
USING ...
1
vote
0answers
29 views
Processing data with different schema
I am using Azure Databricks's AutoLoader.
I have blob storage with lots of JSON files. Those consists of a few dozen of different schemas.
The current solution is based on inferring the schema and ...
0
votes
0answers
10 views
Logging Attached Cluster Information in Databricks / Spark
I would like to do some performance testing on Databricks. To do this I would like to log what cluster (VM type e.g. Standard_DS3_v2) I was using during the test (we can assume that the driver and ...
0
votes
1answer
26 views
Azure databricks data frame count generates error com.databricks.sql.io.FileReadException: Error while reading file abfss:REDACTED_LOCAL_PART
I have this pyspark script in Azure databricks notebook:
import argparse
from pyspark.sql.types import StructType
from pyspark.sql.types import StringType
spark.conf.set(
"fs....
1
vote
1answer
35 views
Set content type when uploading to Azure Blob Storage via Databricks
I am uploading a static site using the databricks platform specifically using the below command for pushing html content to a location.
dbutils.fs.put("/mnt/$web/index.html", html, overwrite=...
1
vote
1answer
161 views
ModuleNotFoundError: No module named 'PIL' when I want to import sparkdl in databricks
I am trying to implement a deep learning pipeline, I need to import sparkdl package in databricks (community edition).
My other installed libraries include:
spark-deep-learning:1.4.0-spark2.4-s_2.11,
...
0
votes
0answers
11 views
DBFS mount on custom Databricks container missing
Usually on a Databricks provisioned cluster, the DBFS is visible under the folder /dbfs. We are using a custom Databricks image as described in the Databricks documentation. However this container ...
0
votes
1answer
19 views
What prevents Databricks IP access lists from being executed
I would like to control from which IP addresses can users access a Databricks instance running on our company Azure.
To do this I have set IP access list policy using Databricks REST API documented ...
0
votes
1answer
35 views
Functions triggered by Eventhub
We have an existing solution where there is an Eventhub with IP whitelisting enabled. This eventhub ingests real time event data. We also have a databricks instance within a VNET reading from this ...
0
votes
2answers
193 views
azure datalake gen2 databricks ACLs permissions
I am trying to understand, why my ACL permissions are not working properly in Databricks.
Scenario: I have 2 Users. one with full permissions on FileSystem and. other without any permissions.
I ...
2
votes
1answer
34 views
Databricks Delta tables vs SQL Server Delta table
Is there is a difference between Sql Delta table and Databricks Delta table? It looks like for SQL we use the name on a conceptual basis. The table that stores the difference of Base table is Delta. ...
0
votes
1answer
43 views
Using a JAR from a private Artifacts Feed in Data Factory Databricks task
My goal is to first publish JAR files of my Scala project to Azure DevOps Artifacts Feed and then use those JARs as part of my Databricks tasks in Azure Data Factory. I'm assuming I would have to use ...
2
votes
0answers
28 views
Databricks: OSError: [Errno 22] Invalid argument
I am processing some CSV files in databricks. Most of them are quite small(up to 500 MB).However, some of them have UTF-16 encoding, so I encode them into UTF-8 before reading with Spark. Other thing ...
0
votes
0answers
24 views
Refresh powerbi dataset using Python with Service Principal in Data bricks
Hi Am trying to refresh the powerbi dataset using Service Principal authentication in data bricks.
Did anyone used service principal to refresh powerbi dataset using python from data bricks please let ...
2
votes
1answer
58 views
Databricks - how is number of partitions determined?
I'm processing a file for which I run a repartition(48, key) prior to saving as a delta. At initial run, it generates around 25 partitions within the delta (no issue as it's possible the key resulted ...
-1
votes
1answer
37 views
How to run scala code inside databricks community edition notebook? [closed]
Databricks community edition notebook allows one to practice scala guide and try examples in real-time. However, the default is set to pySpark and I was wondering how I can run scala code instead?
1
vote
1answer
71 views
delta lake in databricks - a consistent “view” of just the last half hour of a stream
I have consisently updated table from spark structured streaming (kafka source)
Written like this (in eachBatch)
parsedDf \
.select("somefield", "anotherField",'partition',...
2
votes
1answer
31 views
azure pyspark from modules import myfunctions; No module name
I have tried a number of methods to import a local script containing an bunch of shared functions from our shared team directory with example in the code below. I also tried "from . import ...
0
votes
0answers
21 views
databricks/R -— Error: Max iterations (100) reached for batch Resolution
I'm working on databricks and using R (I'm new with databricks). I encountered this error:
org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Max iterations (100) reached for batch ...
1
vote
1answer
64 views
Can we connect sql server databases using SPN in databricks using scala?
My use case is to connect to the SQL server using SPN credentials in Databricks notebook. Since I don't have username password and can't generate access token so I have to do it with the help of SPN ...
1
vote
1answer
322 views
How to trigger azure Databricks notebook from Apache Airflow
I have created some ETL in Azure data bricks notebook.
Now trying to execute that notebook from the airflow-1.10.10.
If anyone can help it would be great.
Thanks In Advance.
0
votes
2answers
37 views
Get Maximum from column with Python in Databricks notebook
I use DataBricks as a service on Azure. I need to get maximum value of one field by Python. I wrote below code but I am wondering if there is another way to get max number without Loop.
%python
...
1
vote
1answer
1k views
Parameter to notebook - widget not defined error
I have passed a parameter from JobScheduling to Databricks Notebook and tried capturing it inside the python Notebook using dbutils.widgets.get ().
When I run the scheduled job, I get the error "No ...
2
votes
1answer
39 views
Delta lake RESTORE issue (Databricks)
I am working on Azure databricks and created a delta table on ADLS Gen2.
I have 4 versions of delta lake created already.
I am trying to restore to version number 2 with the following command.
%sql
...
1
vote
1answer
36 views
Replace values of each array in pyspark dataframe array column by their corresponding ids
I have a pyspark.sql dataframe that looks like this:
id
name
refs
1
A
B, C ,D
2
B
A
3
C
A, B
I'm trying to build a function that replaces the values of each array in ref by the corresponding ...
0
votes
0answers
21 views
How to visualise the complex data on EMR just like we visualise it on Databricks using display() method
I'm writing a pyspark job and I want to visualise the complex data on EMR just like we do on databricks using display which helps in testing.
-5
votes
1answer
218 views
Convert Scala Code to Python for Reporting
I have this code in Scala and not massively familiar with Python to be able to convert it:
val formatterComma = java.text.NumberFormat.getIntegerInstance
def createTD(value: String) : String = {
...
1
vote
1answer
36 views
Delta lake in databricks - creating a table for existing storage
I currently have an append table in databricks (spark 3, databricks 7.5)
parsedDf \
.select("somefield", "anotherField",'partition', 'offset') \
.write \
....
0
votes
1answer
32 views
pyspark 'Index' object is not callable
This is a copy and paste line from a data science experiment I am trying to run.
for col,num in zip(df.toPandas().describe().columns(),range(1,11)):
It shows output but not on Databricks.
Error is:
...
-1
votes
1answer
48 views
Pyspark for converting a column String to Date
Im trying to convert a "dob" column from string to date datatype to do some basic operations in pyspark. My Input is
long_name age dob wage_eur
Cristiano ...
0
votes
1answer
51 views
Can't read a VCF file through spark
I am trying to read a vcf file using spark.
Spark 3.0
spark.read.format("com.databricks.vcf").load("vcfFilePath")
Error:
java.lang.ClassNotFoundException: Failed to find data ...
-1
votes
2answers
297 views
What is my file-system-name and storage-account-name? And how do I find it?
I am trying to connect databricks to my blob containers in azure data lake gen2.
I can't find what my file-system-name is or my storage-account-name is anywhere for a connection.
dbutils.fs.ls("...
0
votes
0answers
47 views
Databricks: Running DBSCAN clustering in the same notebook as a Random Forest model wildly changes model predictions
TLDR: Fitting DBSCAN in a Databricks notebook greatly changes performance of classification model being fit later in the same notebook. No idea why.
I'm using a pyspark notebook within an Azure ...
0
votes
0answers
30 views
How to get the list of nodes running for my spark job in Databricks
I'm trying to run a Pyspark Script in Databricks where that Databricks Cluster is being used by many users including me...
So my question here is, will I be able to obtain (or) is there a way to get ...
0
votes
2answers
36 views
Using DataBricks API 2.0 with Tokens
I'm trying to hit DataBricks API 2.0 using Bearer Tokens and I'm getting 200 response but results are not showing.
I'm running this command,
curl -H @{'Authorization' = 'Bearer <Token Here>'} ...
0
votes
4answers
1k views
why can't I delete all files from an azure blobstorage container in databricks
I want to remove all files from a blobstorage container using:
dbutils.fs.rm
Pointing to one specific file like this works fine:
dbutils.fs.rm("/mnt/inbox/InvLog.txt", True)
But I want to ...
-1
votes
1answer
55 views
How to properly separate columns
I'm having trouble with Spark SQL. I tried to import a CSV file into spark DB. My columns are separated by semicolons. I have tried to separate the columns by using sep to do so, but to my dismay, the ...
-1
votes
0answers
48 views
How to select row that contains multiple variables in a dataframe
I have a data frame like this:
date
x
y
01/01/1995
Two
Three
01/02/1995
Two
Three
01/03/1995
Two
Three
01/04/1995
Two
Three
01/05/1995
Two
Three
01/06/1995
Two
Three
01/07/1995
Two
Three
01/...
-1
votes
0answers
35 views
Horovod timeline error in databricks(not able to write the horovod activity to file)
I am trying to record horovod activity using horovod-timeline in databricks.
I am using the following code
timeline_dir = "/dbfs/ml/horovod-timeline/"
os.makedirs(timeline_dir)
os.environ['...
1
vote
0answers
29 views
Databricks ConvertToDelta - Parquet table to Delta - “AssertionError: assertion failed: File name collisions found”
sampleDf = spark.createDataFrame([(1, 'A', 2021, 1, 5),(1, 'B', 2021, 1, 6),(1, 'C', 2021, 1, 7),],['msg_id', 'msg', 'year', 'month', 'day'])
sampleDf.show()
sampleDf.write.format("parquet")....
0
votes
0answers
31 views
Pyspark: how do we insert into a table in SQL Server database using data frame rows
When I'm reading data from Microsoft SQL Server using Databricks notebook by using following command:
EXAMPLE:
query="(select Id,Abbreviation from dbo.temp) abc"
df_St = spark.read.jdbc(url=...
12
votes
6answers
9k views
Azure Databricks - Can not create the managed table The associated location already exists
I have the following problem in Azure Databricks. Sometimes when I try to save a DataFrame as a managed table:
SomeData_df.write.mode('overwrite').saveAsTable("SomeData")
I get the following error:
...