Questions tagged [azure-data-factory]

Microsoft Azure Data Factory is a cloud service used to invoke (orchestrate) other Azure services in a controlled way using the concept of time slices. Data factories are predominately developed using hand crafted JSON, this provides the tool with instructions on what activities to perform. While still in preview, the introduction of Azure Data Factory V2 allows for an easier data integration experience using the api and not only JSON files.

Filter by
Sorted by
Tagged with
0 votes
0 answers
5 views

ADF input variables monitoring

I have an Azure Data Factory (ADF) where I run a master pipeline that invokes sub-pipelines using an input parameter. The sub-pipeline queries a table, and the table name is passed as an input ...
Karol Bakiera's user avatar
0 votes
0 answers
6 views

Change data capture - Azure SQl Database to On premise NOSQL Database

I've Azure SQL database that I want to capture real time data changes from the database and ingest into my on premise NOSQL database. I am planning to enable Change Data Capture (CDC) in azure SQL ...
user2813702's user avatar
0 votes
0 answers
9 views

Throwing an error code: 9603 while creating a ADF linked service for Redshift

While creating a linked service with using Redshif it shows an connection failed with the below error: Error Code: 9603 ERROR [08S01] [Microsoft][Amazon Redshift] (10) Error occurred while trying to ...
Vaishali Gouli's user avatar
0 votes
0 answers
11 views

Combining multiple files in azure data factory using data flow's union activity

Hello I have quite a few files that I want to combine using Azure Data Factory. I'm currently using Union activity in dataflow to combine a few files at once but I have about a thousand files and was ...
pandasandnumpy's user avatar
-2 votes
0 answers
12 views

SCD2 implementation for datawarehouse tables [closed]

I have 3181 tables in my staging database(i.e Azure SQL database) I want to do SCD2 implementation for all the tables in my target azure SQL database (i.e data warehouse) by using azure data factory. ...
SUSHANTH SANJU's user avatar
0 votes
0 answers
20 views

select tableschema and tablename from a json variable and copy data from these tables of Azure SQL to Blob Storage

I have to create a variable that contain json like this: { "tables": [{ "tableschema":"temp1", "tablename":"prova1", ...
Giusy's user avatar
  • 1
0 votes
0 answers
22 views

Alternative way of writing complex ETLs in Synapse Analytics

We have a Synapse Analytics workspace where we have a number of simple data flows and pipelines which perform simple transformations (ETLs, ELTs). Usually, they read from a data lake (ADLS Gen2) and ...
user2128702's user avatar
  • 2,059
-1 votes
1 answer
44 views

Valid phone number field in Azure Data factory

I am having customer data in csv file and in that I have one column phone number. I need to clean the phone number field. I am getting the phone number column value like below 98377657 (valid phone ...
Sam's user avatar
  • 9
0 votes
1 answer
51 views

How to use the derived column and column pattern in adf to divide multiple columns by 4?

How to use the derived column and column pattern in adf to divide multiple columns by 4? I have multiple columns that start with the characters: 'test' All these columns have to be divided by 4. I ...
Herwini's user avatar
  • 375
0 votes
3 answers
51 views

Make pipeline succeed even after single iteration on failing in copy activity

I am coping data from blob storage to SQL database. In source I have multiple files with different schema so when schema doesn't match to destination table it fails copy data for that file. I want ...
Sumit's user avatar
  • 1
0 votes
1 answer
46 views

Data Factory Copy Upsert doesn' work for Synapse with varchar(max)

I think it is quite common situation like I have in my pipeline. I load data to stage table in Synapse, it is small dimension table with heap index settings and I wanted to use option in Data Factory ...
Alexander's user avatar
  • 347
0 votes
1 answer
50 views

Type conversion settings missed in Azure Data Factory in Copy activity

I would like to use option allowDataTruncation from Type conversion settings in Copy activity in Azure Data Factory, but this option is missed. In my pipeline, in one copy activity source(Postgres), ...
Alexander's user avatar
  • 347
1 vote
1 answer
72 views

notebook activity in ADF not able to call another synapse notebook

Synapse Notebook reference - Call Synapse notebook from another with parameters notebook_name = '/NB_dim_load' using mssparkutils.notebook.run(notebook_name, 90, {"notebookparams": ...
Madhu Tallapaneni's user avatar
0 votes
0 answers
24 views

Missing nodes during copy data pipelines [closed]

I am using Azure Data Factory to copy some data from one database to another. I have 18 nodes that I copy for 12 tables, and 10 of them work perfectly however the last two have random missing nodes ...
Sina Sen's user avatar
0 votes
0 answers
28 views

How to copy ADF pipeline to different new branch without affecting main branch?

We have a main branch that we are using to publish to prod which contains all of the working and WIP pipelines. Now we would like to separate these WIP pipelines to a different branch, since there ...
n_himawari's user avatar
0 votes
1 answer
48 views

Why does my import projection in adf dataflow rename the columns in a weird format?

Why does my import projection in adf dataflow rename the columns in a weird format? While resetting the schema of the data flow source infers the columns as follows (this is also the desired outcome ...
Herwini's user avatar
  • 375
0 votes
1 answer
23 views

Azure Data Factory - Exception occurred when converting value from type 'String' to type 'Single'

I created an ADF pipeline that has as a Source a CSV file and as a Sink SQL Managed Instance. There is one column called "Use" and this column is of type "Real" in the SQL MI. ...
Olgaraa's user avatar
  • 69
0 votes
1 answer
33 views

Databricks Pyspark- Change excel column format from general to number

I am writing a xlsx file using df.write.format("com.crealytics.spark.excel").option("header",True).mode("overwrite").save(outputPath) and using df.withColumn() to change ...
Apoorv Tomar's user avatar
0 votes
0 answers
28 views

Azure Data Factory - Data Flow - Graph API Sink throttling [closed]

Data Flow runs into throttling Graph issue. How to handle 429 Error without erroring the Dataflow? https://learn.microsoft.com/en-us/graph/throttling#sample-response I am using the Request interval (...
AVRM's user avatar
  • 1
0 votes
0 answers
19 views

Azure Data Factory - updating Node to 20 in Azure Devops pipeline

I have a pipeline created according to: https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-improvements I tried to update Node version to 20.x: - task: NodeTool@0 ...
gerwaz's user avatar
  • 83
0 votes
1 answer
55 views

Combining files in Azure Data Factory using Copy data activity causes rows to be mixed up in sink dataset

I used a copy data activity in my pipeline to merge the files in my ADLS folder. However, in the sink dataset, the 2 folders' rows are mixed up eg instead of A1,A2,A3,B1,B2,B3 I am getting something ...
pandasandnumpy's user avatar
0 votes
1 answer
70 views

Error loading dataset in Azure Data Factory

I have a dataset that i can see it in a preview: But, when I create a source in a dataflow, and i choose that dataset, the flow doesn´t import the rows, only the columns name: Please, someone can ...
Luis Gustavo Martinez Osorio's user avatar
0 votes
0 answers
33 views

Azure copy activity (ingest data)

Attempt to ingest file using azure data factory activity (copy activity) from a website. This is the link: https://portaldatransparencia.gov.br/download-de-dados/receitas/2023. The pipeline fails ...
Hugo Araújo's user avatar
1 vote
1 answer
37 views

Reshape JSON response in Powery Query

I am trying to use Power Query to get data from an API. However, I think Power Query is having difficulty with the schema. I can't find the right steps in PowerQuery to make this a flat table ...
Mike Tomaino's user avatar
0 votes
0 answers
21 views

Prevent data corruption in azure mapping data flow

We're ingesting thousands of XML files, one example of this being a shipment event recon. In the data flow we're splitting up the contents of the file to be sinked to various tables, e.g. Order ...
Jacques's user avatar
  • 6,915
0 votes
1 answer
61 views

Data flow errors when importing projection and adding sink in Azure Data Factory

I am implementing cache sink in ADF however, I am having issue when importing the projection and adding the sink, there will be error when adding the sink, but when I delete the sink and import the ...
NgoVanDuc's user avatar
1 vote
1 answer
44 views

Historical data population using data change history from SalesForce

I have the below 2 tables already being ingested to my SQL Server from Salesforce. I have no control over the storage in Salesforce. 1. LiveTable contains the latest data. All changes are updated in ...
Raging Bull's user avatar
  • 18.6k
0 votes
0 answers
36 views

Dataflow in Azure Synapse refuses to be cloned directly: “Error while saving entities. Details: RangeError: Maximum call stack size exceeded”

The Dataflow in question consists of the following elements: 2x Source 1x Sink 4x Derived Column 2x Union 1x Select 2x Filter 2x Aggregate 4x Flowlet: referencing the same Flowlet, consisting of: -- ...
gCW's user avatar
  • 1
0 votes
1 answer
48 views

Selecting columns base on names in Azure Data Factory's dataflow activity

I have some columns in my dataset that I want to map. I want to select all these columns to map to my sink dataset except for the column named 'Total People' I know I need to use the select activity ...
pandasandnumpy's user avatar
0 votes
1 answer
33 views

ADF foreach on JSON Object

I have a json configuration file which is of type json that has a total of two objects something similar to this. [ { "Student": A "Class" : 8 "Homework: : { "type": ...
Sarah's user avatar
  • 1,221
0 votes
1 answer
44 views

Using select activity to remove column from source dataset that has dynamic filepath

In in my dataflow, I have a source that references a dataset whos file path references a pipeline variable I also have a select activity that I use to select the columns I want to map to my sink data ...
pandasandnumpy's user avatar
0 votes
1 answer
58 views

How To Parse JSON Case Insensitive In Azure Data Factory

I am new to Azure Data Factory and we have the following 4 different formats of JSON data stored in the (source) SQL Database: Source SQL Table (Having JSON Payload) RequestID Endpoint Payload 0001 /...
SauGo's user avatar
  • 3
0 votes
1 answer
34 views

Is there any equivalent of ADF copy activity in databricks?

I want to copy data into databricks from different external sources such as azure data lake, blob store, SQL databases, NOSQL databases etc. Is there any way we can do this databricks just like we do ...
MUHAMMAD UMER's user avatar
0 votes
1 answer
21 views

How to Convert SSIS Package xml to Azure Data Factory xml Format?

Hi there i have SSIS packages and i want to move jobs to Azure Data Factory pipelines. I dont mean execute SSIS package. Im looking for a easy way to move packages activities to adf pipeline. All ...
padawan's user avatar
  • 21
0 votes
1 answer
48 views

How to pass pipeline variable into dataflow source dataset

I have a pipeline variable that contains the string "customers.txt" I want to pass it into my dataflow's source dataset's file path dataflow's source dataset However, it is not showing up. ...
pandasandnumpy's user avatar
0 votes
1 answer
37 views

Load percentage value along with percentile from excel file to SQL table using ADF

I have few percentage values in a Excel file. Now I want to load the data into a SQL table using Azure Data factory copy activity, While loading the data into a SQL table from Excel file the ...
Sandi Poojitha's user avatar
0 votes
1 answer
32 views

Unsure how to use a ForEach Activity in a Data Pipeline to iterate over rows in a 2 column dataset derived from an API Json document

PROBLEM CONTEXT I have successfully connected to an API and sent a request to an endpoint that returns rows of data as a Json array. This Json document is subsequently saved as a Json Dataset for ...
Michael Minto's user avatar
0 votes
1 answer
50 views

how to remove carriage return on Merged JSON Files using Azure Data Factory (ADF)

My requirement is to merge multiple JSON files as single JSON file using ADF tool and also remove multiple nested arrays and make it as single array. sample Json file 1st file data: 2nd file data: ...
RAMA RAJU's user avatar
0 votes
0 answers
83 views
+50

Data flow swapping schema name with table name

I have a pipeline that calls a data flow in Azure, and I've had to rebuild it twice and the same thing has started happening again. Somehow when the data flow runs its taking the parameter I created ...
Jacques's user avatar
  • 6,915
0 votes
1 answer
40 views

ADF Copy Activity Issue

I am using Copy Activity in ADF to copy data from SQL. But In one of the table's column there is an HTML code so when the data get copied in CSV file the HTML code got spread across other rows. My ...
Saksham Agarwal's user avatar
0 votes
0 answers
32 views

ADF specific requirement encountered I needed to interact with two REST APIs in Azure DevOps (AzDO) with authentication: 'User Assign Manage Identity [closed]

I would like to share a specific requirement I encountered in Azure Data Factory (ADF) and the solution I implemented, which is currently working seamlessly. In this scenario, I needed to interact ...
Haribansh Gupta's user avatar
0 votes
0 answers
20 views

Azure Data Factory File System Linked Service Not Closing Session [closed]

We have a pipeline in ADF that pulls CSV file data from an internal file share and have recently had some issues with one particular file. It typically looks to be corrupted, but today I was informed ...
Zebediah Jenkins's user avatar
0 votes
0 answers
53 views

Incrementally load data in parquet file using azure data factory copy activity

I have to read the parquet file from ADLS gen 2 and put back to the data into azure SQL DB. I tried to load the data normally with copy activity, it is working fine, when source of the storage parquet ...
Developer KE's user avatar
0 votes
0 answers
23 views

How do i access and read avro file using http linked service in azure data factory

I am trying to read avro file located in internet location using azure data factory copy activity. I tried creating pipeline with httplinked service and source dataset as avro, i am getting following ...
akshindesnowflake's user avatar
0 votes
0 answers
24 views

How to Load Data from Microsoft Business Central into SAP Analytics Cloud via Live Connection? [closed]

I am working on a project where I need to integrate data from Microsoft Business Central into SAP Analytics Cloud. I want to establish a live connection for real-time data updates. Is there a direct ...
Elinehal's user avatar
0 votes
1 answer
38 views

Using pipeline variables inside derived column in azure dataflow

Hi I have a variable in my pipeline that I would like to reference in my dataflow. I tried the steps listed here but my final result is an empty column. Azure Data Factory - Can we use local variables ...
pandasandnumpy's user avatar
0 votes
0 answers
39 views

Unable to import .pem file in Azure Key Vault

I want to upload .pem file in Azure Key vault and want to access the secret from ADF. There are 3 options available: Upload it under Keys in Azure Key Vault- But keys are not accessible from ADF ...
Ritasree Sen's user avatar
0 votes
1 answer
48 views

Dataset connection in Azure Databricks

I have published a dataset in Azure Data Factory but there is no way I can access the dataset in the databricks. The dataset was published from a service that is connected to AWS S3. Here's the ...
slinger's user avatar
-1 votes
2 answers
40 views

Backslash automatically getting removed from the json while reading and need to preseve the backslash - Azure Data Factory

Backslash from the json is getting removed after the file is processed by copy activity I have json input file which contains the below object INPUT { "Name": "xyz", "...
Nagarajan Lakshmi Narayanan's user avatar
0 votes
1 answer
62 views

Json merged file parsed in csv file in ADF Copy data activity

I am retrieve some API response which are saved in separated files through for each loop. Then I am using a copy data activity where all the JSON files created in for each loop, I am merging in one ...
user12309502's user avatar

1
2 3 4 5
171