1

I'm working on a SSIS package in VS 2010 for a 2012 instance. I'm importing flat files from a vendor who will not clean their data. I cannot get beyond the "Flat File Source" step because the data is so mangled it hangs and won't continue parsing.

Here is an example of good data with headers:

EventID|AccountID|ListID|ID|Date
1|3000|20030|1092997696|10-Nov-2014 09:36:13

Here is bad data that will (and is) be/ing captured by error handling:

1|3000|20030|1092997696;ҧ��DAVNJ��|11-Nov-2014 06:40:28

Here is data that hangs my package:

1|3000|20030|1092997696ci[
a5��~[�t:RW�uXXïA,u��ïn��I�    �JA!QXQ|11-Nov-2014 08:27:27

How do I deal with this? Remember, I cannot get beyond the flat file parse step to use a derived column/conditional split/script task.

Thanks in Advance! Kirsten

3
  • It appears you have a carriage return in there. The SSIS parsing library is going to stop at that row and report back that you had a short line.
    – billinkc
    Commented Nov 11, 2014 at 21:19
  • 1
    In a case like this, I'd be inclined to write a file preprocessor in "technology of your choice" to translate the bad files into a something that SSIS isn't going to puke on. I know, double processing, what's the point of using the tool if it can't do this, etc, etc. I'm just trying to offer an approach based on the limitations.
    – billinkc
    Commented Nov 11, 2014 at 21:21
  • 1
    I agree with billinkc. To make a recommendation on how to go about the preprocessing, we'd need to know a bit more about the constraints. Are there any rules that are guaranteed to be followed, even when data is bad? For example, will pipes only appear as delimiters and always appear when they should? If so, you could probably clean the file up pretty easily with a C#/VB script task. Commented Nov 11, 2014 at 21:24

1 Answer 1

1

Figured out a way to get the data in so I can clean it!

CREATE TABLE dbo.crap_data_varcharmax(
    DataBlob NVARCHAR(MAX));

BULK INSERT dbo.crap_data_varcharmax
FROM '\\SQLSERVERNAME01\e$\Folder\FileName.txt' WITH (ROWTERMINATOR = '\n', FIELDTERMINATOR = '|', FIRSTROW = 2);

SELECT * 
FROM dbo.crap_data_varcharmax;

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.