3

I'm an intern in charge of researching an azure project.
At the moment i'm devising an architecture for a part of the project.

The goal is to convert multiple csv files to an SQL database in the cloud. These csv's will be sent from random locations in the country and need to be processed so the database can eventually be accessed using a web service.

I'm totally new to the azure scene and have been schooling myself but it's all a bit fuzzy in my head.

some info:

The csv's are small files but about 20.000 would be received daily yes it needs to be SQL storage because we need to aggregate the data easily.

what will be in the csv and needs to be stored??
a unique key value (string)
a consumption value (double)
a datetime stamp (datetime/string)
a quality value (int)

The architecture i had in mind would be:
Http requests to the cloud (does the cloud need a listener service?)
A queue service that holds the csv's before they are processed
The sql drive storage (direct import? or do i need some kind of worker role in between?)
A web service that will get requests from an external AOS or a client application with a query for the data in the sqlDB.

Am i correct in assuming this problem can be solved with standard components or do i need to implement a vm role? How would you set this up?

Any input would be much appreciated because i really feel lost in the clouds :)
I hope I gave a clear overview of the requirements...
It's not easy explaining something you don't fully grasp yourself

2 Answers 2

3

You don't need a VM role at all. Here's a strawman idea:

  • Set up a web service that lets you push your csv files up (easy to do in a Web Role with an svc). Have that service method store each csv in an Azure Blob in some specific container (like "uploads"), with a name like "guid.csv" - just call Guid.NewGuid().ToString() to generate a guid on the fly. Once that's done, create a queue message referencing that file name.
  • In the Run() method of either the same role instance hosting your svc (just override Run() ) or in a separate worker role, set up a while(true) loop to just read from the queue to grab a csv needing import, reading the blob into a memory stream and dumping to a temp file on disk, and then calling a local helper method to parse the csv and call the SQL Insert.
  • Set up another Web Service for retrieving data. Again, this can either be hosted in the same web role or another.

Absolutely no need for VM Role.

3
  • I realize I didn't mention that your SQL would go to SQL Azure, not a vm-hosted SQL Server setup. SQL Azure is a substantial subset of SQL Server, and you should have no issues using it for the type of storage you're doing. Apr 7, 2011 at 18:04
  • Hi, thanks for the reply! I spent all yesterday afternoon thinking about a setup and it came close to what you listed in the reply. Seems sturdy enough so i'll have a go at it and see if I run into trouble or not. I was wondering if it is possible to skip the blob storage and just have a worker role retrieve the csv's and store them directly in a queue? (Since the csv's will never exceed the 8kb max) Again, loving the input here! tx
    – peanut
    Apr 8, 2011 at 8:02
  • You "could" keep them in the queue and skip blob storage. However, that could be limiting later on, especially if you ever wanted to re-examine your csv's (or reprocess them a different way, for instance). If you stick with the blob technique, you'll also be able to read several queue messages at once; if you store your csv with the message, you'll limit that ability. Apr 8, 2011 at 17:23
0

Is there a reason why you can not just use BCP (Bulk Copy) to import the data directly into SQL Azure? BCP supports CSV files and I suspect you could create a pretty simple process to import the data on a daily basis using this tool. If you do this, make sure you read up on some of the ways that you can optimize the load of the data. This can really make a difference if you have large data sets.

1
  • thank you for the outstanding link, it will come in handy. The reason i can't use a bulk copy is because i do not control the instances that send the csv's. these are fixed points that send data on a regular basis (i'm sorry i can't get into detail about this :))
    – peanut
    Apr 8, 2011 at 7:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.