How are we doing? Please help us improve Microsoft Azure. Take our short survey

Questions tagged [azure-data-factory-2]

Microsoft Azure's cloud data integration service, it can compose data storage, movement, and processing services into automated data pipelines. It can connect to a wide variety of sources, not only from Azure. Its version 2 features more graphic tools and SDKs, providing an easier user experience compared to its previous version.

Filter by
Sorted by
Tagged with
0 votes
1 answer
20 views

Can't use Az.Datafactory in Powershell because my factory is the wrong resource type

I'm trying to use Az.Datafactory module in Powershell to interact with my data factory. I've created this factory in the Azure portal. But the module can't seem to find the factory, because it's ...
0 votes
0 answers
11 views

How can i enable ssl for an oracle linked service in azure data factory?

I followed this link to configure my Oracle linked service - Oracle 19c database We are using Oracle connection manager between SHIR and Oracle database. We are also using alternate servers in oracle ...
0 votes
1 answer
24 views

Unioning using Wild card character * in Azure Data Flow

My File 1 has a,b,c 1,2,3 File 2 has e 23 I am doing the Wildcard path as "*.txt" My Expected Result should be : a,b,c,e 1,2,3,NULL NULL,NULL,NULL,23 But it came as : a,b,c 1,2,3 23,NULL,...
0 votes
0 answers
19 views

How to auto import new DAGs in Azure Data Factory Managed Airflow?

Once we imported files from a Container in Storage, it would make sense to auto import if any file is modified or excluded or if there is a new file in the Container. But currently this job is ...
  • 187
0 votes
1 answer
37 views

Saving pyspark dataframe from azure databricks to azure data lake in a specific folder created dynamically

I am doing some ETL process in Azure. 1. Source data is in Azure data lake 2. Processing it in Azure databricks 3. Loading the output dataframe in Azure data lake to a specific folder considering ...
  • 9
0 votes
1 answer
24 views

How to store 10000 character string in a Column ,Does ADF reads it or truncates it in Dataflow in the sink

I have a column called Notes Where user will enter their long notes which can go greater than 10000 characters ,they store it in col, in csv . If I read that file in Dataflow and sink to blob as csv ...
0 votes
2 answers
55 views

Azure Data Factory: How to check if a folder exists

I've setup a Cost Management export that is trigged weekly to a Storage Account. In that storage account the CSV is exported inside a container, a folder, another folder, and then inside a folder with ...
0 votes
0 answers
28 views

Deafult IR is used in ADF [closed]

My copy activity when on run is using the default IR provided by the ADF. Though the source and sink datasets of the copy activity is pointed to a different IR. I have a metadata activity before the ...
0 votes
1 answer
44 views

Azure Data factory, Data flow output to be uploaded to FTP

Below is my dataflow. My requirement is 1)To merge 2 files. 2)Create a merged tab file 3) Upload the file to FTP I was able to achieve first 2 steps. Now gettibg error while trying to upload to FTP....
0 votes
1 answer
25 views

Can we install ADF SHIR and Logic Apps on-prem Data Gateway on the same machine?

We understand that a VM on-prem hosts the SHIR and this then is registered to the ADF. Can we use same VM for installing the on-prem Data Gateway for Logic apps or both need to be on separate machines?...
0 votes
1 answer
30 views

Can we cap throughput in ADF copy activity?

Is there any way to cap throughput in ADF copy activity like we have in AzCopy ? This is needed to ensure our Ingestion ( parallel Ingestion of all tables of a given source) doesn't consume all the ...
0 votes
1 answer
51 views

I Need to Add few columns Based on Condition in pyspark

One File has Status_MP_c This column need to Split in same file and need to make sent,open,click columns if it is true will you provide logic in pyspark
  • 17
0 votes
1 answer
30 views

Can we push Azure blob data to Tableau hyper file in Azure data factory

I have to push my blob csvs,parquets to Tableau .Generally Tableau hyperfiles we would be creating in Tableau. I dont find any connector for Tableau. Is there any other way to do it via python code or ...
0 votes
1 answer
49 views

How to get the first day of the month in Azure data flow

I have "Date" column with yyyy-MM-dd format .I want to fetch the 1st day i.e, if date is 2023-05-28 ----> 2023-05-01 I have tried method below and apply it in iif(toString(Date)>=...
1 vote
1 answer
44 views

I have created a DataFow in Azure Data Factory. In the sink , I need to add a time stamp to my filename

In the sink filename, I need to add filename SampleFileName_dev_8Mar2023.tab In the Dataflow - Sink - Settings - Filename , I need to add the Dataflow expression builder. What expression do I need to ...
0 votes
1 answer
52 views

Azure Data Factory Remove Empty Child Directory

In the following file structure container/Directory/Sub-Directory I wish to delete Sub-Directory --because at this point in my process it is empty. Note: I am using the term 'Directory' as opposed to ...
  • 35
0 votes
1 answer
49 views

How to do not in ( ) concept in Azure data flow

I have given an expression like not(in(["FIX","ANGELA","ZERO","$0","INSTALLED",toString(null())],upper(CUSTOMER_PO_NUMBER))in(["FIX","...
0 votes
0 answers
64 views

How to pass a pipeline variable into a Dataset Parameter?

I'm currently working on a pipeline which requires me to dynamically create a @FileName(FileName_YYYY_MM_DD.csv) variable which has the date appended to it. The aim is to export data from an Azure SQL ...
0 votes
1 answer
46 views

Beat methods to validate/test the output data with original data

I have built dataflows from customer input data in Asure data factory. We designed the Workflows which are built on Alteryx(another Etl tool) in Azure data factory . Now testing the sample data is ok ...
0 votes
1 answer
65 views

Zero Padding in a variable

Below is my scenario : Input value=1234 Input Range= 2 Expected output =1200 Input value=12345 Input Range= 3 Expected output =12000 The ask is replace the number of characters/digits from right with ...
  • 3,349
0 votes
1 answer
62 views

Getting Error while sending parameters from child Pipeline to Inner Parent Pipeline in ADF

I referred https://learn.microsoft.com/en-us/azure/data-factory/tutorial-pipeline-return-value I created one Parent Pipeline ->2 nd Parent Pipeline ->Child Pipeline I declared one variable ...
-1 votes
1 answer
60 views

Can I add an Azure data factory service principal inside Azure Databricks to grant it access to run notebooks rather than giving it "Contributor" role

I followed this link to use MSI authentication for Axure Data factory to configure and run my Azure Databricks notebooks. This requires ADF managed identity to have "Contributor" permissions....
0 votes
1 answer
35 views

Wrong columns while importing schema from Excell sheet in Azure Data flow

My input file is Excel which is on DatalakeGen2 blob.Am using Excel connector in Azure data flow and specified all the filename,path,cotainer .But its showing me the wrong Columns When i import schema ...
0 votes
0 answers
49 views

Connection string format for Oracle 19c database with secondary server in Azure Data Factory [closed]

We are planning to use complete connectionString of an Oracle database as a secret stored in the key vault and use this in our Linked Service. I followed this link to get this understanding. Original ...
0 votes
1 answer
86 views

How to obtain the array element as string

I have a JSON result in my copy job which like this: { "data": [{ "id": 1, "name": "Company 1" }, { &...
  • 361
0 votes
1 answer
94 views

How to retrieve objects out of Json in Azure Data Factory

I have a Json file that when run by Data Factory only contains one nested object "Properties" and it is fine but Value4 looks like another nested object and is being visible as one long ...
  • 29
0 votes
1 answer
107 views

Trigger Azure Data Factory Pipeline using Postman

I'm kinda lost, I'm trying to trigger a pipeline manually using the following: POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft....
  • 113
0 votes
1 answer
58 views

How to authenticate to Azure Active Directory Protected API from a scheduled Azure Data Factory pipeline

We are using Microsoft Identity Platform for our Authentication. We have registered our .NET API (e.g. name of the api = example-api) in Azure Active Directory App registrations. We are planning to ...
2 votes
0 answers
71 views

Deployment changing the integration runtime of synapse and data factory artifacts to AutoResolveIntegrationRuntime

I'm running into a scenario when I run my deployment to different environments even though I have SelfHostedIntegrationRuntime specified on the linked services it auto resolves to ...
  • 927
0 votes
1 answer
76 views

ADF dataflow external library use case

I'm using ADF dataflow to process my data. I have an use case where I need to convert a column X to Y based on custom logic in a library. When handling in code, we could import the library into the ...
  • 109
0 votes
1 answer
23 views

automate deploy datafactory ARM template

I am trying to automate the deploiement of azure datafactory arm template I am following this https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-improvements ...
0 votes
1 answer
58 views

Excel file Processing using Azure Data factory

There are 10 excel files in on premise machine and each excel file contains different number of sheets and the sheets will vary dynamically(mostly increasing the sheets).For example Excel-1 contains ...
0 votes
1 answer
48 views

Azure Data Factory can not get multiselect option list from dynamics 365

Azure Data Factory Link Service has a Dataverse connector that allows you to extract data from Dynamics 365. After extracting data we noticed that none of the multi-select fields were populating our ...
  • 1,497
0 votes
1 answer
71 views

Azure Python SDK for DataFactory to point to a specific GIT branch

I'm trying to update the Datasets programatically using azure-mgmt-datafactory SDK. I'm able to connect to the DataFactory, list and update datasets to the adf_publish/live mode. However, I'm ...
0 votes
1 answer
60 views

How Parameterize Truncating a table in Azure SQL DB within an Azure Data Factory Copy Activity

I following an SO question on how to truncate a table in Azure SQL DB with ADF how to replace data in azure sql database using azure data factory? I am trying to emulate the sample in the question ...
  • 1,497
0 votes
1 answer
39 views

Azure data factory - Concurrence problem in the child pipeline

i have a parent and child pipelines. Parent pipeline triggers the child pipeline inside the foreach activity. I have totally 6 items in the array so foreach triggers the child pipeline 6 times. I made ...
0 votes
1 answer
43 views

How to construct a FETCH query to Fetch all fields in Dynamic CRM table with Azure Data Factory

I would like help fetching a table in Dynamics CRM with Azure Data Factory I use the following parameters to select all fields from a regular table @concat('SELECT * FROM ',pipeline().parameters....
  • 1,497
1 vote
1 answer
68 views

Can we make the ADF JSON mapping in COPY Activity Case-insensitive?

So, actually I'm using the ADF Copy Activity to load the CSV files from the azure blob storage to snowflake table. And column mapping is done using the json with source and sink column name as below: {...
0 votes
1 answer
49 views

Restrict Non-Owners from changing the Collaboration branch of Azure Data Factory using Azure Policy

We would like to impose the following restrictions on Azure Data Factory Don't let users change the collaboration branch from develop to anything else. Allow only Owners to change the collaboration ...
  • 1,125
-1 votes
1 answer
53 views

How to use secure string parameter in Add dymamic content

I have one parameter created which is of type "SecureString" . I want to use that directly in any AddDynamiccontent box . But it throws me the error like its ecpecting the type string but ...
0 votes
3 answers
73 views

Azure Data Factory copy activity with json data

I am using a copy activity with request method as POST to copy the JSON response from a rest api. The rest api requires a body which contains date and time details in the below format [ { "...
  • 1
0 votes
2 answers
49 views

Convert to required date format with date datatype

I have input col as 12/03/08 viz dd/MM/yy . So I beed it in the format of dd/MM/yyyy . SO I have use the below transformation as : toString(toDate(col,'dd/MM/yy'),'dd/MM/yyyy') This works fine. But at ...
0 votes
1 answer
70 views

Can I use Azure Data Factory for ETL Testing as well?

If I have to do data migration and ETL testing for data in Azure SQL DB's then can I use Azure Data Factory? If yes the please provide some links explaining how? or some tutorials or page where I can ...
1 vote
4 answers
97 views

How to convert YYYYMM to YYYY-MM-01 using Azure data factory expression

I need to convert the YYYYMM to YYYY-MM-01 using azure data factory pipeline expression. I tried the below expression but it giving me error that date value should follow the ISO 8601 format. @...
  • 439
0 votes
1 answer
71 views

How to grab column value from ADLS gen 2 csv file and use the column value in the body of the email,also send blob data as attachment to outlook mail

Here is my Scenario, There will be a drop of csv file into blob storage every day ,that will be processed by my dataflow in ADF and generate a csv in output folder. Now Using logic apps, I need to ...
0 votes
2 answers
46 views

How to copy blob data from one storage acc in one subscription to other storage acc in other subscription using ADF

I have data in one storage acc in dev subscription. I need to move the data to other storage acc in other subscription. Is there any way to do this by ADF by copy activity?
0 votes
0 answers
42 views

Connect Self hosted IR via Private end point from ADF

We have a Azure VM Windows img installed.I need to use that Vm as Self hosted IR , I need to enable the Private end point option in ADF Networking tab. When I foloowed the genneral process like: ...
0 votes
0 answers
40 views

How to have combined activities as one for "Rerun from failed activity" in ADF

Our scenario is that ADF doesn't provide the direct activity for us to manage outer service (spark job) and thus we need to have several activities as one to manage it. Thus the activities would ...
0 votes
1 answer
32 views

ADF CopyData Is it possible to have a dynamic Additional Columns that can be nullable?

I have a configuration table with file names, destination table name (and other configs) to copy data into a SQL table. Sometimes I want the filename in a new column, but not for every files. Is it ...

1
2 3 4 5
55