3

I have several vehicles that send a data to server each minute. The server should be listening and decode the data to store in the database. There will be thousands of entries per minute. What is the best approach to solve that problem?

2
  • also bear in mind that the vehicle may not have a data connection to your server when it thinks it needs to update your server - as such, you might want to look into handling situations where you cannot update (do you wait, do you abort?) Mar 31, 2009 at 20:29
  • Hiee paul were you able to solve your problem, please reply because i am developing a GPS server & i need help. I cant figure out where to start. Aug 24, 2015 at 6:09

3 Answers 3

6

My personal favorite, WCF or WebService farm pumps the data to a Microsoft Message Queue (MSMQ) and have a application server (1 or more) convert the data and put it into the DB.

As you get deeper (if you ever need to), you can use the features of MSMQ to handle timeouts, load buffering, 'dead-letters', server failures, whatever. Consider this article.

On the web facing side of this, because it is stateless and thin you can easily scale out this layer without thinking about complex load balancing. You can use DNS load balancing to start and then move to a better solution when you need it.

As a further note, by using MSMQ, you can also see how far 'behind' the system is by looking at how many messages are in the queue. If that number is near 0, then you good. If that number keeps rising non-stop, you need more performance (add another application server).

10
  • Thanks for help... I never heard about that MSMQ ... The performance is good? And Dont I need to implement a multithreading system?
    – Paul
    Mar 31, 2009 at 20:33
  • 1
    This is really what MSMQ was built for. The performance is very good. I have a system using it that the communication overhead of the above is about 25ms (separate machines each) from request to web to MSMQ to application to DB. Mar 31, 2009 at 20:37
  • As for multithreaded, not required, but can help performance a bit. You can create a object that reads off the MSMQ to do processing and then instantiate a copy on one thread per CPU if you wanted. Mar 31, 2009 at 20:39
  • I´m reading MSMQ tutorial, and It said: "Message queuing must be installed on both the sending (client) and receiving machine (server)." How can it works that way? My vehile device is a simple hardware with a GPRS device to send the data...
    – Paul
    Mar 31, 2009 at 20:54
  • 1
    Have the device send the data (via WCF, SMS, Web Services, whatever) to server somewhere that puts it into MSMQ. This is also AMAZINGLY safer security wise as your DB can be non-public facing. Apr 1, 2009 at 2:11
1

We're doing exactly what Jason says, except using a direct TCP/UDP socket listener with a custom payload for higher performance.

1
  • 1
    Aye, and with UDP you can get performance at an unheard of level if you don't care about missing a call every once in a while. That is why alot of MMORPG games use custom UDP protocols. Apr 1, 2009 at 2:13
0

How long do you expect each operation to take? From what you're saying it seems like you can just write the data straight to the db after processing, so you don't have to synchronize your threads at all (The db should have that taken care of for you).

2
  • I just need to decode the data (fast operation) and save to DB... It can be to only 1000 vehicle or to 10000...
    – Paul
    Mar 31, 2009 at 20:23
  • It sounds like your transmission rate is pretty slow (GPRS) compared to your processing time, so why even multithread at all? You'll just be spending most of the time waiting for more data. In any case, your reciever should have a buffer already that should be emptied pretty quickly. Mar 31, 2009 at 21:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.