2

I'm writing an application that crawls over a long list of links , downloads pages, searching for html elements using xpath queries and stores some of the retrieved info in mysql database. I use multi-threaded solution to get the most of my servers and eliminate the effect of latency.

I write most of my applications using csharp and java, and my web applications I write with asp.net/c#.

What I want to ask is whether or not Node.js is something worth considering from performance view point? given the throughput is the most important factor. Node.js being more portable and cross platform is another reason, but performance is more important for me.

2
  • My initial thought is that yes - node is worth considering for this sort of task. The big selling point for node is performance. It's really designed to handle concurrency - even though its single threaded. I've done multi-threading in C# and its a nightmare.
    – Steve Mc
    Jun 30, 2013 at 19:06
  • It uses call_back which is implemented using thread pool. I'm not sure it is single threaded. I think it is multi threaded because the call back is executed by a thread pool; still you don't need to explicitly make it concurrent.
    – Adi Omari
    Jul 1, 2013 at 21:02

1 Answer 1

0

I think the bottleneck in your application is on the network (HTTP or MySQL), rather than on the code.

Up or down, Node.js is single-threaded and based on message queue, when all "messages" are completed, it quit. Every I/O is waited on thread pool (unless you use sync methods, which is not encouraged).

Few things on your case:

  • You don't have simple and full controls on processors/threads, but you can always limit number of simultaneous job yourself
  • You don't need to worry about concurrency because there is no concurrency at all
  • All APIs in Node.js are designed with streamlined I/O in mind, you may only write 20-50% of code (in terms of lines) to write a web crawler, than in C#
  • Node.js has no XML library built-in, you can find a few on NPM
  • JavaScript is an interpreted language, code is slower than C#. But since your are more on I/O, I think it shouldn't slow you down too much
  • Debugging JavaScript on Node.js is painful, but web crawling should not be very complicated either

I have wrote a few simple web crawlers on Node.js and would definitely recommend you to give it a try.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.