4

My app is running on App Engine, but I would like to access its NDB DataStore entities from my Compute Engine VM to do some processing and write the results back to the App Engine DataStore. How can I do that?

Also, are the Google Cloud DataStore and App Engine DataStore the same thing? https://developers.google.com/datastore/ https://developers.google.com/appengine/docs/python/ndb/

4 Answers 4

9

David's solution requires you to use App Engine instance time to make requests, but you can bypass it and make requests directly to Datastore from Compute Engine instance. There is a pretty good tutorial about how to do this. But its not so pretty like ndb.

>>> import googledatastore as datastore
>>> datastore.set_options(dataset='project-id')
>>> req = datastore.BeginTransactionRequest()
>>> datastore.begin_transaction(req)
<datastore.datastore_v1_pb2.BeginTransactionResponse object at ...>
3
  • Isn't the doc for Google Cloud DataStore? According to David's answer I took it as the Cloud DataStore is a standalone version of App Engine DataStore...or is it that by giving the Dataset-ID == project-id it points to the App Engine DataStore? Sep 1, 2014 at 10:50
  • Clod datastore == app engine datastore. Sep 1, 2014 at 11:17
  • 8
    "Google Cloud Datastore for accessing an existing App Engine Datastore from Compute Engine": developers.google.com/datastore/docs/… Sep 1, 2014 at 11:19
3

Google Cloud Datastore is kind of a standalone version of App Engine's datastore.

Back to your problem, you have two options :

  1. Write a web-service to expose your entities from the App Engine app to the Compute Engine VMs. One option is Cloud Endpoints. Cloud Endpoints uses OAuth2 authentication and the Compute Engine VMs are shipped with OAuth2 service accounts that you can use to authenticate to the service.

  2. Use the App Engine remote API to access the Datastore. The remote API provides access to the Datastore as if you were in an App Engine instance. But by default the API requires you to provide the password of an App Engine amdin, so you might have to store your password in the Compute Engine VMs, which is not safe.

2

Aside from options that @David explained, you can also look into Managed VMs: they are a cross between Compute Engine and App Engine - basically a managed Compute Engine instance that has access to App Engine services.

1
  • Yeah, but I think at this point there is no local development server story for Python, apparently you have to deploy to test it...so is the best practice to change the version, deploy and then if all is well you change the version back to prod and redeploy? Sep 1, 2014 at 10:45
1

The officially recommended way is to use the datastore client libraries; see https://cloud.google.com/datastore/docs/reference/libraries You need to create a service account, or use the standard compute engine service account, give permission to either all APIs, or to datastore to that service account, and create the compute engine instance to be part of that service account. See here for more information. Then you can do something like:

from google.auth import compute_engine                                                                                      
from google.cloud import datastore                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                

datastore_client = datastore.Client(project='yourproject',     
                                    credentials=compute_engine.Credentials())                                                                                                                                                                  
q = datastore_client.query(kind='YourEntity')                                                                                    
q.add_filter('field_name', '=', 'HelloThere') 
print list(q.fetch(1))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.