7

Is there a project which helps setup a Hadoop cluster on office desktops, when they are idle?

I'd like to experiment with Hadoop/MR/hbase but don't have acces to 5-10 computers. The computers at work are idle after hours and are connected to each other through a very high speed connection. What's more, data on these computers stays within our network so there is no privacy issue.

In order for this to work I need a fairly light weight monitor running on each machine. When the computer has been idle for X hours, it will join the cluster. If the user logs on, it has to drop out of the cluster and return all CPU/memory back.

Does something like this exist?

1
  • I highly recommend Amazon AWS / Elastic Map Reduce. You can set up fairly powerful Hadoop clusters quickly.
    – Suman
    Apr 24, 2012 at 21:59

5 Answers 5

3

You can use task scheduler to detect idle state and then start/stop a hadoop vm with virtual box or vmplayer. Or you can write a powershell script that does start stop based on resource usage.

2
  • Wow, I had no idea windows scheduler could schedule with respect to idle time! Others ma find this useful: microsoft.com/resources/documentation/windows/xp/all/proddocs/…
    – Shahbaz
    Apr 14, 2012 at 22:09
  • Do we have any references or link where we can establish and use virtual box in each of the Desktops which are having 4 Gigs of RAM and setup Hadoop Cluster. It would be a good experimental lab. Any thoughts or ideas? Jun 25, 2013 at 23:39
1

Hadoop is not a computation grid it is a more a data grid (see slide 9 in this presentation). The point is that with hadoop that data is spread over the cluster and thus the data has to be stored on the computers. The time it would take to copy the data over/remove it when they're not idle would probably not be worth it - you'd be better off using hadoop in the cloud (amazon,Azure etc.)

1
  • I won't be removing data. Shutting down a node when the computer is being used by a human just means that I will temporarily remove the node from the hadoop cluster. When the computer is available to hadoop, the node will start up and re-join the cluster.
    – Shahbaz
    Apr 17, 2012 at 13:38
1

I would use something like Condor: http://research.cs.wisc.edu/condor/

2
  • Thanks HaloWebMaster. I already have a Condor solution at my university (greedy.epfl.ch). It is great, but I would like to use something like HDFS, so I am looking for is a Hadoop implementation.
    – Renaud
    Jun 21, 2012 at 17:19
  • You could perhaps use the Parrot virtual file system in the Condor jobs.
    – mhucka
    Jun 5, 2016 at 23:56
1

You might want to take a look at Virginia Tech's Project Moon http://www.wired.com/wiredenterprise/2012/05/project_moon/

2
  • Is their system available outside Virginia Tech? I could not find a link to download it...
    – Renaud
    Jun 21, 2012 at 19:23
  • Renaud, you might want to contact the Lead PI of Project Moon Wu-Chun Feng people.cs.vt.edu/~feng to check whether its available for download
    – fjxx
    Jun 22, 2012 at 20:06
0

Look at solutions like NEREUS which is a good MPC solution in Java

1
  • Thanks Grooveek. I already have a similar solution at my university (greedy.epfl.ch). What I am looking for is a Hadoop implementation.
    – Renaud
    Jun 15, 2012 at 13:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.