I have an application that will generate a log entry for every search query it processes (from Solr) so that I can calculate certain statistics for my search engine. For instance the number of queries without results, the average number of hits, et cetera. Now I'm wondering how best to perform this analysis. Load is estimated at a high of ten thousands searches per day with statistics generated over a weekly period. In other words, I'm looking for the best way to calculate statistics on up to one hundred thousand XML files.
The process will be controlled by Apache Camel and I'm currently thinking that XQuery will be my best bet of tackling this problem. As I'm still working on establishing the schema I can't run any real world tests so I wanted to garner some opinions on the best approach to take before I dive in. Some questions:
- Can XQuery handle this many files or will I need to transform them all into a single document with XSLT?
- Is XQuery the right tool for the job? It seems to me like it would be more efficient than trying to do this in a high-level programming language while XSLT would be too low-level.
- One alternative may be to index these queries in Apache Lucene/Solr. Would this be more efficient?
- Can I get away with storing these XML files on a file system? Or do I need to load them into an XML database? (Which I'm not familiar with.)