1

I would like to extend the data model of a remote database that is available via a web service interface. Data can be requested via HTTP GET and is delivered as JSON (example request). Other formats are supported as well.

// URL of the example request.
http://data.wien.gv.at/daten/wfs?service=WFS&request=GetFeature&version=1.1.0&typeName=ogdwien:BAUMOGD&srsName=EPSG:4326&outputFormat=json&maxfeatures=5

First object of the JSON answer.

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
       "id": "BAUMOGD.3390628",
       "geometry": {
          "type": "Point",
          "coordinates": [
             16.352910973544105,
             48.143425569989326
          ]
       },
       "geometry_name": "SHAPE",
       "properties": {
          "BAUMNUMMER": "1022 ",
          "GEBIET": "Strassen",
          "STRASSE": "Jochen-Rindt-Strasse",
          "ART": "Gleditsia triacanthos (Lederhülsenbaum)",
          "PFLANZJAHR": 1995,
          "STAMMUMFANG": 94,
          "KRONENDURCHMESSER": 9,
          "BAUMHOEHE": 11
       }
    },
...

My idea is to extend the data model (e.g. add a text field) on my own server and therefore mirror the database somehow. I stumbled into CouchDB and its document-based architecture which feels suitable to handle those aforementioned JSON objects. Now, I ask for advise on how to replicate the foreign database initially and on a regularly basis.

Do you think CouchDB is a good choice? I also thought about MongoDB. If possible, I would like to avoid building a full Rails backend to setup the replication. What do you recommend?

7
  • CouchDB does replication out of the box and uses a simple REST API. That being said I love MongoDB, but I imagine it will be a little fancier to sync up your models. Apr 25, 2012 at 23:56
  • You don't own the remote db, do you? Apr 25, 2012 at 23:56
  • @JamundFerguson I saw the replication feature in CouchDB. How would you setup a the replication with the particular example? Please be more descriptive.
    – JJD
    Apr 26, 2012 at 0:19
  • @SergioTulentsev No, I do not own the remove database. All I can access is their public API / web service interface.
    – JJD
    Apr 26, 2012 at 0:20
  • 1
    Replication can be set up only between two CouchDB instances. Not between a CouchDB and some 3rd-party web service. Apr 26, 2012 at 0:20

1 Answer 1

0

If the remote database is static (data doesn't change), then it could work. You just have to find a way to iterate all records. Once you figured that out, the rest is simple as a pie: 1) query data; 2) store the response in a local database; 3) modify as you see fit.

If remote data changes, you'll have many troubles going this way (you'll have to re-sync in the same fashion every once in a while). What I'd do instead is create a local database with only new fields and a reference to the original piece of data. That is, when you request data from remote service, you also look if you have something in the local db and merge those two before processing the final result.

3
  • The remote database will change. Objects can be added, modified or deleted. Even they could change the attributes. What stays is the ID of the objects, I guess. From a programmer's point of view I would subclass/wrap their objects and add custom fields. Although, I need to retrieve updates from them from time to time - I do not want to forward a frontend end request to their database every time.
    – JJD
    Apr 26, 2012 at 0:17
  • Well, it's up to you. You need a way to get/generate ids of all existing objects in their database so that you can go ahead and fetch the objects. Apr 26, 2012 at 0:25
  • The iteration will be the hard part. I am even not sure whether I could request by ID. I had the idea to request quadrants of data since they all have geolocations.
    – JJD
    Apr 26, 2012 at 0:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.