10

I have a child pipeline that receives instructions from a control file using a dataset. These instructions define what directory to copy files from.

First this child pipeline checks that a file exists via Get Metadata activity on the source folder. It then executes a child pipeline to process the data if one or more childitems are returned from GetMetaData.

In the control dataset, there is also a required Y/N field meaning I can ignore the error if the folder or file does not exist.

If the folder does not exist, the GetMEtadata will fail. If it exists but no files, I get 0 child items. So 2 different things happen for file or folder missing (an error or 0 items).

In either case, I need to route the output of GetMetaData to an IF that checks if the file was required. If not, consume the error and return. If it was required, raise an error.

I can't find a way to raise an error though. Just as important, is there an alternative approach that would work or fit better with ADF V2 design?

Many thanks, Mark.

6 Answers 6

12

I use a Set Variable Task to achieve this.

In the variable expression, I divide by zero. This throws an error which propigates up my pipeline(s) as needed. Most generic way I could think to do it.

"name": "Throw Error",
"description": "You can't throw an error in DF so I try to divide by zero. :)",
"type": "SetVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
   "variableName": "ThrowError",
   "value": {
      "value": "@div(1,0)",
      "type": "Expression"
   }
}
1
  • 2
    Interesting approach. The advantage of the accepted answer is that you can write some info that will get recorded in the log. With this solution, I think it comes down to how you name the variable. That said, it is a simple work-around to a missing feature.
    – MarkD
    Oct 2, 2019 at 4:38
7

I have the similar scenario and I manage that with SqlStoreProcedure.

"storedProcedureName": "sp_executesql", "storedProcedureParameters": { "stmt": { "value": "Declare @err_msg NVARCHAR(150)SET @err_msg=N'Error occurred in this pipeline somehow somewhere something. Best regards. EXISTSCheers'RAISERROR(@err_msg,15,1)", "type": "string" } }

StoredProcedureName: sp_executesql

StoredProcedureParameter: stmt

Value: Declare @err_msg NVARCHAR (150) SET @err_msg=N'Error occurred in this pipeline somehow somewhere something. Best regards. EXISTSCheers'RAISERROR (@err_msg,15,1) (of course you can change error text:)

Type: string

2
  • 1
    Thank you - I had also thought of calling an azure function which returns 500. For this one, a sql connection required and any sql connection errors would cause a 'throw within a throw'. That said, if we have a sql connection, this suggestion is an easy work-around and I'll mark as answer. n.b. I added votes and a comment to this User Voice ticket to help raise this in the product backlog. feedback.azure.com/forums/270578-data-factory/suggestions/….
    – MarkD
    Oct 16, 2018 at 16:58
  • Yes of course, this is only a workaround. Also, if you work with ADLA you can throw an exception. I am not aware of any generic work around for this, and we need it, so i will also upvote this feedback. Cheers. :)
    – DraganB
    Oct 16, 2018 at 18:35
3

I have found the easiest way to throw an error is to do a SQL lookup on the following query:

THROW 99999, 'Error message thrown', 1

For us, we make extensive use of Azure Function Apps, so we created a special function called "ThrowError" that throws an error with a message provided as a parameter. That is the easiest only if you already use Azure Function Apps, not worth it to just create the one function.

3

In November 2021 ADF gained the Fail activity, in the General group.

It has two specific properties, Fail message and Error code, both of which are dynamic. They behave exactly as would be expected.

When executed it always has the status Failed. The message and code are evaluated and appear in the pipeline output.

1
  • I know it's an old question, but this should be now the accepted answer (and it's available in Synapse Pipelines as well!).
    – Arklur
    Oct 11, 2022 at 4:22
0

The solution I am using is to create a mock API in API Management that returns 500 HTTP code together with an error message. More details on how to do this can be found here: https://learn.microsoft.com/en-us/azure/api-management/mock-api-responses.

From inside the ADF pipeline, I call this mock API using Web Activity passing in the error message.

Then I can see the failed pipeline with expected error.

0

I use a Web Activity and GET a URL that doesn't exist: www.notreal-nvlsuifhg9348h4932.com.

(I do like Chris Hume's Set 1/0 option though. Might steal that ;) )

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.