0

In a perfect world, I'd have a bunch of data readily available to me without any time spent asking for and receiving it. But in the context of real applications, like google or facebook, you have a mountain of data stored in a database that requires time to query, and then you're trying process that data in order to draw meaningful conclusions / relationships.

In the context of counting and sorting lots of data in sql, you'd store data in summary tables to avoid the processing... and just update those tables with cron. But statistical analysis and nlp seems to be different.

The question is, at what point in the lifespan of data should the actually statistical/nlp/etc analysis occur?

2 Answers 2

1

The way you usually do this is collect data, have it some sort of database (SQL or NoSQL) and then for processing dump it into a hadoop grid if it's a huge amount of data; otherwise do whatever you usually do. Then you have jobs analyzing that data and feeding the results back to you.

Get data -> Store it -> Dump it -> Analyze it -> Use results of offline analysis

Data crunching on an actual database just doesn't work too well.

3
  • sure, but presumably the analysis is pretty small, maybe a statistical model or a vector of weights you can apply to new data.
    – nflacco
    Apr 5, 2012 at 3:00
  • lets assume we're talking about a large analysis -- the point of the question is really about when the appropriate time is to do a major analysis. and how to do it without it being on the fly. think google's search "index"
    – Kristian
    Apr 5, 2012 at 13:33
  • You aren't doing it at index time, that's for sure. Let's say you are a big search engine- how many milliseconds do you think you have during index time to devote to NLP? You usually do this as a hadoop job once a day or once a week and then use that information to inform user queries into your index.
    – nflacco
    Apr 6, 2012 at 2:10
0

Depends what you have in mind when you say NLP. The moment a few tens of tweets/status updates stored somewhere you can start reading and analysing them. It's probably not a great idea to repeatedly query your only production server while the NLP is taking place- you might want to take a dump of whatever data there are and work from there.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.