35

I am redirecting rows from a flat file source to a flat file destination. The default metadata in the redirected rows are:

  • The original flat file source row
  • The ErrorCode
  • The ErrorColumn

What I get in the output flat file is the source row (nice) and the error code (not nice, ex. -1071628249) and the error column (not nice since it's the internal ID of the column).

How can I transform the rows to output the error message (e.g. "The data was truncated.") and the column name as defined in the flat file source?

In other words, instead of ...,-1071607675,10 I'd like to see:

...,The data was truncated,Firstname

or alternatively (if the previous is not possible);

...,DTS_E_FLATFILESOURCEADAPTERSTATIC_TRUNCATED,Firstname.

10 Answers 10

21

Error message list is in the following location: MSDN, Integration Services Error and Message Reference https://learn.microsoft.com/en-us/sql/integration-services/integration-services-error-and-message-reference?view=sql-server-ver15

And column Id Number can be found in SSIS's Data Flow Task: select the task component that generates the error, Advanced Editor, 'Input and Output Properties' tab, External Columns Properties.

2
  • For a specific issue, I was able to mash the Error message list page together with Error codes fairly easily using the Power Query Excel Add-In. This will probably become viable within SSIS in SQL 2016. HTH.
    – Mike Honey
    Jul 28, 2015 at 7:56
  • 1
    Do the column IDs in the data conversion output correspond to the ID row in that properties list, the LineageID row, or perhaps something else?
    – user565869
    Feb 16, 2016 at 16:38
16

Part of the question (adding the error description) can be achieved with a script component. This is described in Enhancing an Error Output with the Script Component.

It seems that the Dougbert blog has a solution to adding the column name, but it's far from simple. I'm amazed this is so difficult to do in SSIS; you'd think it was a basic need to know the name of the source and column.

4
  • 3
    This is a link-only answer, and does not describe the solution in the case of link rot or if for some reason the linked site's solution is not readable. My dev machine is not connected to the internet, and I don't have access to another machine on which I can open the package on the Dougbert site.
    – Monty Wild
    Mar 8, 2016 at 0:57
  • 1
    So Bernhard should plagiarize code from other links in the Microsoft Books Online to make this more complete? Jul 14, 2016 at 14:00
  • 2
    "I'm amazed this is so difficult to do in SSIS" – if it wasn't Microsoft SQL Server in question here, I'd be amazed, too. Feb 27, 2018 at 17:17
  • 1
    The Dougbert blog link is broken
    – Mick
    Mar 15, 2019 at 0:29
6

There is a far simpler answer. Simply redirect the error output to a new destination file (CSV or whatever) and then enable a DataViewer on the error output....

Data Viewer in SSIS

1
  • 1
    That's OK if you want to view the errors during development, but if you also want to log them in production you'll need the answer by Satyanath G/Kohei TAMURA.
    – Richard
    Jan 24, 2018 at 17:01
6

It can be achieved using script component as transformation, Redirect error output to the script component and follow the steps to achieve what you are looking for.

(1) Open script component ,

Input Columns select

  1. ErrorColumn
  2. ErrorCode

Input and Output add Output columns

  1. ErrorDescription (DT_STRING 500)
  2. ErrorColumnDescription (DT_STRING 100)

(2) Edit Script

Paste the following code

    using System;
    using System.Data;
    using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
    using Microsoft.SqlServer.Dts.Runtime.Wrapper;
    #endregion

    /// <summary>
    /// This is the class to which to add your code.  Do not change the name, attributes, or parent
    /// of this class.
    /// </summary>
    [Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
    public class ScriptMain : UserComponent
    {

        public override void Input0_ProcessInputRow(Input0Buffer Row)
        {
            var component130 = this.ComponentMetaData as IDTSComponentMetaData130;
            if (component130 != null)
            {
                Row.ErrorDescription = component130.GetErrorDescription(Row.ErrorCode);
                Row.ErrorColumnDescription = component130.GetIdentificationStringByID(Row.ErrorColumn);
            }
        }
3
  • 1
    GetIdentificationStringByID is not recognized
    – BICube
    Aug 25, 2017 at 16:14
  • Input0Buffer class doesn't contain ErrorDescription or ErrorColumnDescription. Should we add this too? Comments on Input0Buffer class definition says don't edit!
    – Libin M
    Sep 5, 2017 at 19:53
  • IDTSComponentMetaData130 is new in SQL Server 2017. Looking for a 2014 solution and can't find one
    – ozz
    Oct 9, 2018 at 10:24
5

Pragmatic Works appears to have an Error Output Description Transform that is a part of the Community Edition (Free) of the Product they call "Task Factory".

The Error Output Description Transform provides the user with a User Interface that can retrieve valuable information such as the ErrorCode, ErrorColumn, ErrorDescription, ComponentName (that generated the error), ColumnName (if known), ColumnType, and ColumnLength.

It also allows you to pass through any input columns to the Error Output. To be honest it is quite handy and has saved me hours of time troubleshooting my SSIS Packages.

2
  • Some clients forbid third-party components of any sort, even freeware.
    – Monty Wild
    Mar 8, 2016 at 0:50
  • 2
    That doesn't mean it's any less valid of an answer for people that can use third party free components. Jul 14, 2016 at 13:58
4

Using SS2016 and above, it is easy: https://www.mssqltips.com/sqlservertip/4066/retrieve-the-column-causing-an-error-in-sql-server-integration-services/

public override void Input0_ProcessInputRow(Input0Buffer Row)
{
    Row.ErrorDescription = this.ComponentMetaData.GetErrorDescription(Row.ErrorCode);
    IDTSComponentMetaData130 componentMetaData = this.ComponentMetaData as IDTSComponentMetaData130;
    Row.ErrorColumnName = componentMetaData.GetIdentificationStringByID(Row.ErrorColumn);
}

For anyone using SQL Server versions before SS2016, here are a couple of reference links for a way to get the Column name:

http://www.andrewleesmith.co.uk/2017/02/24/finding-the-column-name-of-an-ssis-error-output-error-column-id/

which is based on: http://toddmcdermid.blogspot.com/2016/04/finding-column-name-for-errorcolumn.html

I appreciate we aren't supposed to just post links, but this solution is quite convoluted, and I've tried to summarise by pulling info from both Todd and Andrew's blog posts and recreating them here. (thank you to both if you ever read this!)

From Todd's page:

  1. Go to the "Inputs and Outputs" page, and select the "Output 0" node. Change the "SynchronousInputID" property to "None". (This changes the script from synchronous to asynchronous.)
    1. On the same page, open the "Output 0" node and select the "Output Columns" folder. Press the "Add Column" button. Change the "Name" property of this new column to "LineageID".
    2. Press the "Add Column" button again, and change the "DataType" property to "Unicode string [DT_WSTR]", and change the "Name" property to "ColumnName".
    3. Go to the "Script" page, and press the "Edit Script" button. Copy and paste this code into the ScriptMain class (you can delete all other method stubs):
public override void CreateNewOutputRows() {
    IDTSInput100 input = this.ComponentMetaData.InputCollection[0];
    if (input != null)
    {
        IDTSVirtualInput100 vInput = input.GetVirtualInput();
        if (vInput != null)
        {
            foreach (IDTSVirtualInputColumn100 vInputColumn in vInput.VirtualInputColumnCollection)
            {
                Output0Buffer.AddRow();
                Output0Buffer.LineageID = vInputColumn.LineageID;
                Output0Buffer.ColumnName = vInputColumn.Name;
            }
        }
    } }

Feel free to attach a dummy output to that script, with a data viewer, and see what you get. From here, it's "standard engineering" for you ETL gurus. Simply merge join the error output of the failing component with this metadata, and you'll be able to transform the ErrorColumn number into a meaningful column name.

But for those of you that do want to understand what the above script is doing:

  1. It's getting the "first" (and only) input attached to the script component.
    1. It's getting the virtual input related to the input. The "input" is what the script can actually "see" on the input - and since we didn't mark any columns as being "ReadOnly" or "ReadWrite"... that means the input has NO columns. However, the "virtual input" has the complete list of every column that exists, whether or not we've said we're "using" it.
    2. We then loop over all of the "virtual columns" on this virtual input, and for each one...
    3. Get the LineageID and column name, and push them out as a new row on our asynchronous script.

The image and text from Andrew's page helps explain it in a bit more detail:

enter image description here

This map is then merge-joined with the ErrorColumn lineage ID(s) coming down the error path, so that the error information can be appended with the column name(s) from the map. I included a second script component that looks up the error description from the error code, so the error table rows that we see above contain both column names and error descriptions.

The remaining component that needs explaining is the conditional split – this exists just to provide metadata to the script component that creates the map. I created an expression (1 == 0) that always evaluates to false for the “No Rows – Metadata Only” path, so no rows ever travel down it.

Whilst this solution does require the insertion of some additional plumbing within the data flow, we get extremely valuable information logged when errors do occur. So especially when the data flow is running unattended in Production – when we don’t have the tools & techniques available at design time to figure out what’s going wrong – the logging that results gives us much more precise information about what went wrong and why, compared to simply giving us the failed data and leaving us to figure out why it was rejected.

2
    //column error description 
    Row.ErrorDescription = this.ComponentMetaData.GetErrorDescription(Row.ErrorCode);

    //we are getting column name with some extra information
    String rawColumnName = this.ComponentMetaData.GetIdentificationStringByLineageID(Row.ErrorColumn);

    //extracting positions of brackets 
    int bracketPos = rawColumnName.LastIndexOf('[')+1;
    int lastBracketPos = rawColumnName.LastIndexOf(']');

    //extracting column name from the raw column name
    Row.ErrorColName = rawColumnName.Substring(bracketPos, (lastBracketPos - bracketPos));
1
  • 1
    GetIdentificationStringByLineageID is supported in SQL Server 2016+
    – janv8000
    Jul 7, 2016 at 13:27
1

Here is a solution that

  1. Works at package runtime (not pre-populating)
  2. Is automated through a Script Task and Component
  3. Doesn't involve installing new assemblies or custom components
  4. Is nicely BIML compatible

Check out the full solution here.


Here is the short version.

  1. Create 2 Object variables, execsObj and lineageIds
  2. Create Script Task in Control flow, give it ReadWrite access to both variables
  3. Insert the following code into your Script Task
Dictionary<int, string> lineageIds = null;

public void Main()
{
    // Grab the executables so we have to something to iterate over, and initialize our lineageIDs list
    // Why the executables?  Well, SSIS won't let us store a reference to the Package itself...
    Dts.Variables["User::execsObj"].Value = ((Package)Dts.Variables["User::execsObj"].Parent).Executables;
    Dts.Variables["User::lineageIds"].Value = new Dictionary<int, string>();
    lineageIds = (Dictionary<int, string>)Dts.Variables["User::lineageIds"].Value;
    Executables execs = (Executables)Dts.Variables["User::execsObj"].Value;

    ReadExecutables(execs);

    Dts.TaskResult = (int)ScriptResults.Success;
}

private void ReadExecutables(Executables executables)
{
    foreach (Executable pkgExecutable in executables)
    {
        if (object.ReferenceEquals(pkgExecutable.GetType(), typeof(Microsoft.SqlServer.Dts.Runtime.TaskHost)))
        {
            TaskHost pkgExecTaskHost = (TaskHost)pkgExecutable;
            if (pkgExecTaskHost.CreationName.StartsWith("SSIS.Pipeline"))
            {
                ProcessDataFlowTask(pkgExecTaskHost);
            }
        }
        else if (object.ReferenceEquals(pkgExecutable.GetType(), typeof(Microsoft.SqlServer.Dts.Runtime.ForEachLoop)))
        {
            // Recurse into FELCs
            ReadExecutables(((ForEachLoop)pkgExecutable).Executables);
        }
    }
}

private void ProcessDataFlowTask(TaskHost currentDataFlowTask)
{
    MainPipe currentDataFlow = (MainPipe)currentDataFlowTask.InnerObject;
    foreach (IDTSComponentMetaData100 currentComponent in currentDataFlow.ComponentMetaDataCollection)
    {
        // Get the inputs in the component.
        foreach (IDTSInput100 currentInput in currentComponent.InputCollection)
            foreach (IDTSInputColumn100 currentInputColumn in currentInput.InputColumnCollection)
                lineageIds.Add(currentInputColumn.ID, currentInputColumn.Name);

        // Get the outputs in the component.
        foreach (IDTSOutput100 currentOutput in currentComponent.OutputCollection)
            foreach (IDTSOutputColumn100 currentoutputColumn in currentOutput.OutputColumnCollection)
                lineageIds.Add(currentoutputColumn.ID, currentoutputColumn.Name);
    }
}

4. Create Script Component in Dataflow with ReadOnly access to lineageIds and the following code.

public override void Input0_ProcessInputRow(Input0Buffer Row)
  {
      Dictionary<int, string> lineageIds = (Dictionary<int, string>)Variables.lineageIds;

      int? colNum = Row.ErrorColumn;
      if (colNum.HasValue && (lineageIds != null))
      {
          if (lineageIds.ContainsKey(colNum.Value))
              Row.ErrorColumnName = lineageIds[colNum.Value];

          else
              Row.ErrorColumnName = "Row error";
      }
      Row.ErrorDescription = this.ComponentMetaData.GetErrorDescription(Row.ErrorCode);
  }
3
  • 1
    This contains a bug that if two or more executables in the same package have columns with the same lineageID, lineageIds.Add in ProcessDataFlowTask throws an error.
    – Monty Wild
    Mar 8, 2016 at 0:52
  • @MontyWild I put up a "fix" for this at github.com/sorrell/GetErrorColumnNames/issues/1
    – mpag
    May 21, 2018 at 18:11
  • 1
    @mpag, I am not sure what your modified code does differently, but I fixed this by changing LineageID to a dictionary of string, string, and in ProcessFataFlowTask, prepending the executable's ID and a separator character to the column ID. That makes the dictionary key unique to each combination of executable and column id.
    – Monty Wild
    May 22, 2018 at 17:58
1

I connected to the SSIS Error message ref webpage with excel using the get data from web on the data tab. Saved the table in a sheet in excel, then imported it to SQL Server. Then joined it to my error rows table on the decimal code to get the description, and then created a view out of it. Thought this might be useful for those that don't want to mess with the script task.

1

I was pulling my hair for last couple of days. I did everything that is mentioned everywhere but the package/c# was throwing an error. Finally when I decided to give up, I found that my ErrorColumn was coming up as 0 (Zero) because the error was in entire row due to PK/FK constraint violation.

So I modified the script as below:

public override void Input0_ProcessInputRow(Input0Buffer Row)
    {
        Row.ErrorDescription = this.ComponentMetaData.GetErrorDescription(Row.ErrorCode);

        var componentMetaData130 = this.ComponentMetaData as IDTSComponentMetaData130;
        if (componentMetaData130 != null)
        {
            if (Row.ErrorColumn == 0) //Checking if the Column is zero
            {
                Row.ColumnName = "Entire Row. Check PK FK constraints"; //Hardcoded error message
            }
            else
            {
                Row.ColumnName = componentMetaData130.GetIdentificationStringByID(Row.ErrorColumn);
            }
        }
    }

For usual process: https://learn.microsoft.com/en-us/sql/integration-services/extending-packages-scripting-data-flow-script-component-examples/enhancing-an-error-output-with-the-script-component?view=sql-server-2017

Why ErrorColumn value is Zero?: SSIS 2016 - ErrorColumn is 0 (zero)

Hope that helps !!!

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.