4

I'm working on an academic project aimed at studying people behavior.

The project will be divided in three parts:

  1. A program to read the data from some remote sources, and build a local data pool with it.
  2. A program to validate this data pool, and to keep it coherent
  3. A web interface to allow people to read/manipulate the data.

The data consists of a list of people, all with an ID #, and with several characteristics: height, weight, age, ...

I need to easily make groups out of this data (e.g.: all with a given age, or a range of heights) and the data is several TB big (but can reduced in smaller subsets of 2-3 gb).

I have a strong background on the theoretical stuff behind the project, but I'm not a computer scientist. I know java, C and Matlab, and now I'm learning python.

I would like to use python since it seems easy enough and greatly reduce the verbosity of Java. The problem is that I'm wondering how to handle the data pool.

I'm no expert of databases but I guess I need one here. What tools do you think I should use?

Remember that the aim is to implement very advanced mathematical functions on sets of data, thus we want to reduce complexity of source code. Speed is not an issue.

3
  • 1
    You may like to describe more on your data and intended operations on it. As now it seems that you don't need to use relational model. Thanks
    – eat
    Apr 3, 2011 at 11:22
  • @eat: I will mainly do statistical operations: draw graphs (with google charts), perform statistical operation (PCA, critical tests, ..) and study correlation. E.g.: Is there a correlation between height and age? take all heights values and age values and perform a test. Or: what is the total height value? how it changes over age/sex/ ... ?
    – Mascarpone
    Apr 3, 2011 at 13:30
  • 2
    OK, I think pytables and scipy/numpy are right tools then. Thanks
    – eat
    Apr 3, 2011 at 13:36

3 Answers 3

5

Sounds that the main functionality needed can be found from:
pytables
and
scipy/numpy

3

Go with a NoSQL database like MongoDB which is much easier to handle data in such a case than having to learn SQL.

0
1

Since you aren't an expert I recommend you to use mysql database as the backend of storing your data, it's easy to learn and you'll have a capability to query your data using SQL and write your data using python see this MySQL Guide Python-Mysql

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.