In this answer, I will try to give some information and to provide a working example of importing several flat files with different metadata into separate SQL Server tables.
SSIS limitation
Since SSIS requires fixed metadata of the data source and destination, I suggest using a C# code rather than automating the SSIS package creation. If SSIS is required, you should use a Script Task to achieve that task.
In similar cases, I generally use C# or another scripting language such as python.
Why should you not automate the SSIS package generation?
Importing more than 100 flat files to more than 100 tables that should be created at runtime (with no fixed structure) requires creating more than 100 SSIS packages. Even if this process is automated using one of the SSIS automation tools, it is hard to manage and debug a high number of SSIS packages.
Unified SQL Server destination table?
If the destination tables structures mostly have similar structures, it is better to import all flat files into a unified SQL Server destination using a schema mapping approach, you can refer to the following answer:
Solution overview
All code provided in this solution can be improved, but it was written as a prototype
Reading flat-file
I created a class TextImport
that tries to loop over all flat files within a given directory, read the flat-files metadata and store each file data within a System.Data.DataTable
object. (Note that Microsoft.VisualBasic
assembly should be added as a reference to use the TextFieldParser
class)
TextImport txtimp = new TextImport(filename, true, 0); // TextImport(string filename, bool containsHeader, int RowsToSkip)
txtimp.BuildDataTableStructure(); //Read flat file metadata
DataTable dt = txtimp.FillDataTable(); //Convert the flat file to a DataTable object
The TextImport
class tries to detect the file metadata (number of columns, delimiter, text qualifiers, encodings) based on the first 8 lines (arbitrary number) in the flat file.
Creating SQL destination table
Then this System.Data.DataTable
is passed to another class called SQLExport
that generates and executes a CREATE TABLE
statement based on the DataTable
metadata.
SQLExport.CreateDestinationTable(DataTable dt);
Inserting data into SQL
Finally, two insert methods are provided:
- Generate a
INSERT INTO () ... VALUES (),(),()...
statement and execute it to insert data into SQL destination
- Using
System.Data.SqlClient.SqlBulkCopy
class to insert data using BULK INSERT
method.
The main code should look like the following:
static void Main(string[] args)
{
//You should set your database connection string
string connectionstring = @"Data Source=.\SQLINSTANCE;Initial Catalog=tempdb;integrated security=SSPI;";
//This is the schema of the destination table
string Schema = "upload";
//You should set the text files directory
string directory = @"E:\TextFiles";
using (SQLExport sqlExp = new SQLExport(connectionstring, Schema))
{
//if you don't want to traverse subfolders use System.IO.SearchOption.TopDirectoryOnly
foreach (string filename in System.IO.Directory.GetFiles(directory,
"*.txt",System.IO.SearchOption.AllDirectories)){
using(TextImport txtimp = new TextImport(filename, true, 0))
{
txtimp.BuildDataTableStructure();
DataTable dt = txtimp.FillDataTable();
dt.TableName = System.IO.Path.GetFileName(filename);
sqlExp.CreateDestinationTable(dt);
//Insert using BULK INSERT
sqlExp.InsertUsingSQLBulk(dt);
//Creates and Execute an INSERT INTO statment
//sqlExp.InsertIntoDb(dt);
}
}
}
}
Test
I tested this solution on two flat files stored on the following link (the first result of my Google search), as shown in the screenshots below, tables are created successfully, and data is inserted.
Note that all columns are created a NVARCHAR(255)
, you can change this within the SQLExport.cs
file
You can check the complete code in the following GitHub repository (.NET Framework 4.6.1 C# console application):
If you decided to use a Script Task to run this code, you should only copy the classes from the GitHub project into your Script Task project Remember to add Microsoft.VisualBasic
as a reference.
How to run a C# console application from an SQL Server Agent Job?
You can simply edit the C# Console application to pass the connection string and text files directory as arguments, then run it using a cmdExec
job step:
Or even by using an SSIS package:
Alternative - using Task Scheduler
Another alternative is to use Windows Task Scheduler to periodically run the C# console application.
sqlcmd
orbcp
to import the data into the relevant tables.txt
and are on a schema with the file's (base)name? That is a terrible naming convention if so...CREATE
the tables as well, how do you know what the definition of said tables needs to be? Do your files have accompanying format files? How do you know what data types the columns in the new table should have?