-1

I have a SSIS package which is trying to read data from a text file. The issue I am facing is that the text file doesn't have very straight forward data as in it has special characters which are creating trouble

For Example, right after the header row, there's a row full of hyphens, something like -----------------------------------------------------------------------------------------

This SSIS is reading as the first value of the first column beacause of which it fails. How do I get rid of this, without actually removing the row from the file itself?

Also, in later part of the file as well, there are some unwanted rows which I would like to ignore, the format of the file is something like this :

Header

Data

Random Rows

Same header row as above

Data

and so on.....

I would like to know if there's a way to handle this with script task or any other way before or while the 'Flat File source' task gets executed, without actually making changes in the original file.

1 Answer 1

2

I don't know of anyway to filter these rows on input using the Flat File Source component, but you can definitely do some filtering if you read the file in with a Script Component.

If you add a reference to Microsoft.VisualBasic, you can use the below function to read your CSV into a datatable:

public static DataTable ReadInDataFromCSV(string fileName, string delimiter)
{
    DataTable dtOutput = new DataTable();
    //How many lines to read in.  0 for unlimited
    int numberOfLines = 0;

    using (TextFieldParser parser = new TextFieldParser(fileName))
    {
        parser.TextFieldType = FieldType.Delimited;
        parser.SetDelimiters(delimiter);
        //Are column names in first row?
        bool columnNamesInFirstRow = true;
        int rowCounter = 0;
        string[] currentRow;
        while (!parser.EndOfData && rowCounter <= numberOfLines)
        {
            try
            {
                currentRow = parser.ReadFields();
                /*****************************
                Add some kind of logic here to skip over rows you don't
                want to read in
                *****************************/

                if (columnNamesInFirstRow == true)
                {
                    foreach (string column in currentRow)
                    {
                        dtOutput.Columns.Add(column);
                    }
                    columnNamesInFirstRow = false;
                }
                else
                {
                    DataRow dr;
                    dr = dtOutput.NewRow();
                    dr.ItemArray = currentRow;
                    dtOutput.Rows.Add(dr);
                    columnNamesInFirstRow = false;
                }
            }
            catch (Exception e)
            {
                Console.WriteLine(e.Message);
            }

            rowCounter += (numberOfLines == 0) ? 0 : 1;
        }
    }

    return dtOutput;
}

By default, the above code will read a flat file into a DataTable by calling something like:

DataTable myInputData = ReadInDataFromCSV(@"Path to file",",")

If you modify the commend I added inside the try/catch, you can filter out the rows you aren't interested in. For example, to skip the rows with hypens, you can add a simple check like:

if (currentRow.IndexOf("-----") > 0)
{
    continue;
}
else
{
    //If/else statement from the original code that adds the data to a DataRow and then adds it to the DataTable
}

Then you can simply add more similar checks to include/not include certain rows in your file. Good luck!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.