1

Currently I am working with a provider built SQL Server DB. This DB has data coming in from calls made through their system. The main table where data stored has 7 fields. 1 field is the Primary Key then 2 foreign keys, a couple datatime stamps and finally a massive field call "SergmentLog"

Within this field the data comes over unstructured. Here is an example of what the data looks like:

/20160219T154710.554-07/0?S=50&E=3512&CUTC=20160219T155235.662-07&1=100187177120160219&2=0&3=18823&4=user%20queue:icadmin&5=&6=Interact&7=|/20160219T154729.377-07/0?S=50&E=3504&CUTC=20160219T155235.663-07&1=100187177120160219&2=0&3=81592&4=user%20queue:icadmin&5=&6=LocalTransfer&7=%3cDetails%20TransferringUser%3d%22ICadmin%20-%22%20TransferringInteractionId%3d%22100187177120160219%22%20TransferredInteractionId%3d%22100187177120160219%22%20/%3e%0a&8=&9=2|/20160219T154850.970-07/0?S=50&E=3502&CUTC=20160219T155235.663-07&1=100187177120160219&2=0&3=55&4=&5=workgroup%20queue:Central%20Ops%202&6=LocalTransfer&7=%3cDetails%20TransferringUser%3d%22ICadmin%20-%22%20TransferringInteractionId%3d%22100187177120160219%22%20TransferredInteractionId%3d%22100187177120160219%22%20TransferredUser%3d%22Phoenix%20AZ%22%20/%3e%0a|/20160219T154851.025-07/0?S=50&E=3500&CUTC=20160219T155235.664-07&1=100187177120160219&2=0&3=1048&4=&5=&6=Queue&7=%3cDetails%20IVRAppName%3d%22Central%20Ops%202%22%20/%3e%0a|/20160219T154852.073-07/0?S=50&E=3502&CUTC=20160219T155235.664-07&1=100187177120160219&2=0&3=13344&4=&5=workgroup%20queue:Central%20Ops%202&6=Interact&7=|/20160219T154905.417-07/0?S=50&E=3504&CUTC=20160219T155235.664-07&1=100187177120160219&2=0&3=26202&4=user%20queue:icadmin&5=workgroup%20queue:Central%20Ops%202&6=LocalDisconnect&7=&8=&9=5

What I am told is that each "SegmentLog" could have multiple "Events", called "E=" within the SegmentLog field. Each event is broken out by the "|" pipe symbol. But prior to each even, there is a datatime stamp from the server, then a SourceID (called "S=") then finally the EventID (Called "E=")

After each EventID (number from 3500 - 3512) There will be Attributes numbering from 1-9 (Called "1=", "2=" etc.).

Keeping in mind that there may be multiple events per SegmentLog with the same EventID, and not all attributes will show in every EventID (IE E=3502 may only shows attributes 1-6, while E=3503 may show attributes 1-9) what would be the best way to structure this data into a table structure. The tools I have available are building complex search queries inside of a view or intermediate SSIS knowledge.

EDIT

I would like to see the data come out to something like this. But including all attributes:

DateTime                    Sequence  EventID  Attr1                  Attr3  
--------                    --------  -------  -----                  -----
/20160219T154710.554-07/0?  s=50      &E=3512  &1=100187177120160219  &3=18823
/20160219T154729.377-07/0?  S=50      &E=3504  &1=100187177120160219  &3=81592
/20160219T154850.970-07/0?  S=50      &E=3502  &1=100187177120160219  &3=55
/20160219T154851.025-07/0?  S=50      &E=3500  &1=100187177120160219  &3=1048
2
  • Can you please tell me what the datatype of the SegementLog column in your SQL Server table? Adding a create table statement as well may help to clarify this.
    – jymbo
    Feb 25, 2016 at 16:52
  • It is currently nvarchar(max). I am going to write how I would like it to show in an edit to the original post as well. Thank you Feb 25, 2016 at 17:14

1 Answer 1

0

Ok I think this is what you're trying to accomplish.

To test this I added your sample line to a SQL Server table nvarchar(max) column:

if exists (select * from sysobjects where name='BigLongString' and xtype='U')
drop table dbo.BigLongString;
go

create table dbo.BigLongString
( 
 SegmentLog nvarchar(max)
);
go

insert into dbo.BigLongString (SegmentLog)
values ('/20160219T154710.554-07/0?S=50&E=3512&CUTC=20160219T155235.662-07&1=100187177120160219&2=0&3=18823&4=user%20queue:icadmin&5=&6=Interact&7=|/20160219T154729.377-07/0?S=50&E=3504&CUTC=20160219T155235.663-07&1=100187177120160219&2=0&3=81592&4=user%20queue:icadmin&5=&6=LocalTransfer&7=%3cDetails%20TransferringUser%3d%22ICadmin%20-%22%20TransferringInteractionId%3d%22100187177120160219%22%20TransferredInteractionId%3d%22100187177120160219%22%20/%3e%0a&8=&9=2|/20160219T154850.970-07/0?S=50&E=3502&CUTC=20160219T155235.663-07&1=100187177120160219&2=0&3=55&4=&5=workgroup%20queue:Central%20Ops%202&6=LocalTransfer&7=%3cDetails%20TransferringUser%3d%22ICadmin%20-%22%20TransferringInteractionId%3d%22100187177120160219%22%20TransferredInteractionId%3d%22100187177120160219%22%20TransferredUser%3d%22Phoenix%20AZ%22%20/%3e%0a|/20160219T154851.025-07/0?S=50&E=3500&CUTC=20160219T155235.664-07&1=100187177120160219&2=0&3=1048&4=&5=&6=Queue&7=%3cDetails%20IVRAppName%3d%22Central%20Ops%202%22%20/%3e%0a|/20160219T154852.073-07/0?S=50&E=3502&CUTC=20160219T155235.664-07&1=100187177120160219&2=0&3=13344&4=&5=workgroup%20queue:Central%20Ops%202&6=Interact&7=|/20160219T154905.417-07/0?S=50&E=3504&CUTC=20160219T155235.664-07&1=100187177120160219&2=0&3=26202&4=user%20queue:icadmin&5=workgroup%20queue:Central%20Ops%202&6=LocalDisconnect&7=&8=&9=5')
go

I then created a SSIS package to pull this data and parse it. The Data Flow Task looks like this: Data Flow Task
The SQL statement in the OLE DB Source component is:

select 
      SegmentLog 
from 
      dbo.BigLongString;

The Script Component is a transformation, that has an asynchronous output:

Inputs and Outputs Form

If you expand the Output 0 tree you can see all of the columns added. The Attr* columns are all dt_wstr 500. I'm not sure how big these can get, so you might want to change the datatype. The rest of the columns I just make dt_wstr 50:

Output Columns

Here is the code for the Script Component. Make sure you build before exiting out:

 #region Namespaces
 using System;
 using System.Data;
 using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
 using Microsoft.SqlServer.Dts.Runtime.Wrapper;
 using Microsoft.SqlServer.Dts.Pipeline;
 #endregion

 [Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
 public class ScriptMain : UserComponent
 {
   private PipelineBuffer inputBuffer;

 public override void Input0_ProcessInputRow(Input0Buffer Row)
 {

    //length of blob
    int blobLen = 0;
    //the bytes of the blob
    byte[] webBlob = null;

    string webStr = null;

    string[] dateSplit = new string[] { "|" };

    //get blob length. Hardcoded to 0 since we only look at one column
    //in this example
    blobLen = (int)inputBuffer.GetBlobLength(0);

    //gets string from blob, hardcoded columnindex since we only have 1 column
    webStr = ConvertBlobToString((byte[])inputBuffer.GetBlobData(0, 0, blobLen));

    //holds value for dates in string
    string[] dates = webStr.Split(dateSplit, StringSplitOptions.None);

    //Loop through each date
    foreach (string date in dates)
    {
        //Parse out each attribute for a given date
        string[] attributes = date.Split('&');

        Output0Buffer.AddRow();

        //Loop through each attribute in date, you can remove the "&"+ if you do not need these in the values
        for (int i = 0; i < attributes.Length; i++)
        {

            switch (i)
            {
                case 0:
                    Output0Buffer.DateTime = attributes[i].Substring(0, attributes[i].IndexOf('S'));
                    Output0Buffer.Sequence = attributes[i].Substring(attributes[i].IndexOf('S'), attributes[i].Length - attributes[i].IndexOf('S'));
                    break;
                case 1:
                    Output0Buffer.EventID = "&" + attributes[i];
                    break;
                case 2:
                    Output0Buffer.CUTC = "&" + attributes[i];
                    break;
                case 3:
                    Output0Buffer.Attr1 = "&" + attributes[i];
                    break;
                case 4:
                    Output0Buffer.Attr2 = "&" + attributes[i];
                    break;
                case 5:
                    Output0Buffer.Attr3 = "&" + attributes[i];
                    break;
                case 6:
                    Output0Buffer.Attr4 = "&" + attributes[i];
                    break;
                case 7:
                    Output0Buffer.Attr5 = "&" + attributes[i];
                    break;
                case 8:
                    Output0Buffer.Attr6 = "&" + attributes[i];
                    break;
                case 9:
                    Output0Buffer.Attr7 = "&" + attributes[i];
                    break;
                case 10:
                    Output0Buffer.Attr8 = "&" + attributes[i];
                    break;
                case 11:
                    Output0Buffer.Attr9 = "&" + attributes[i];
                    break;
            }
        }

    }
}

public override void ProcessInput(int InputID, Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer Buffer)
{
    inputBuffer = Buffer;
    base.ProcessInput(InputID, Buffer);
}

public string ConvertBlobToString(byte[] webBlob)
{
    //string to return
    string webStr = null;

    //get string from blob
    webStr = System.Text.Encoding.Unicode.GetString(webBlob);

    return webStr;

}

}

Run the package and you should see the data parsed out as you expected in the Data Viewer:

Data Viewer

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.