4

We currently have an appengine app (java) with millions of entities. We do a lot of reporting using the map reduce framework + cron to power dashboards and such.

However, we'd like to have the ability to run adhoc queries over our entire data set. The way we do that now is write a mapreduce, deploy, run mapreduce, see results. We'd like to not have to do the deploy step. That is, just goto some admin interface, specify our query and maybe some custom code to do post processing and then see the results. We'd do a lot more adhoc queries if we didn't have to deploy every time.

Has anyone done something like this? What did you learn? Any good strategies?

1 Answer 1

2

It's a Python example but I'm pretty sure you can do the same with Java. One solution if you simply want to count entities by filters. You can create a mapper which handle filters from mapreduce.yaml

- name: Query on Actors
  mapper:
    handler: mapper_api.query_process
    input_reader: google.appengine.ext.mapreduce.input_readers.DatastoreInputReader
    params:
    - name: entity_kind
      value: common.models.Actor

    - name: filters
      value: age<27, name=toto

Then in your mapper_api.py You have to explode and handle every filter:

def query_process(entity):
  ctx = context.get()
  pms = ctx.mapreduce_spec.mapper.params

  filters = pms['filters']

  if match(entity, filters):
    yield op.counters.Increment("matched")

So now in your /mapreduce you can choose the mapper Query on Actors and pass to it some filters.

1
  • I don't think you can do this in Java. You have to iterate over the entire data set. I'm fine with this, so I may have to write code that takes in these filters as strings, converts them to code dynamically to see if the entity matches the filter. This has to have been done before....
    – aloo
    Mar 4, 2012 at 8:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.