3

We're planning to migrate away from a series of static config files related to each client deployment.

All client data lives in MySQL, the meta-data for a client deployment lives in a static text file: what database/shard to use, the repository directory/location for the file store, and a bunch more info such as default pagination, rates and enabled modules.

We want to move away from these static config files—which are more than just key-value pairs at the moment and make use of Perl's hashes and arrays, but could be simplified that way with some effort—and make use of something which can live in a fast-responding database, LDAP or other repository. The flexible structures of LDAP—which allow nesting/hierarchy of data I think—look appealing, but I was wondering whether there was some other advice about the best way to do this I don't think LDAP was really designed for this, and it has been around for a long while.

The "key" we would use to identify a deployment, database, repository path and other variables will be the "domain" of the deployment (these will be unique), and ideally we'd like the config storage solution to be a central point or federated service that multiple servers can query very quickly (over a LAN).

Very few "update" actions will be undertaken on this data, yet the read rates will be very very very frequent, so speed of read is critical.

Any advice?

Geoff

2
  • Interesting. I just realised who you are, after posting my answer.
    – Noon Silk
    Sep 21, 2009 at 12:11
  • Yeah, I wondered the same ;-) Thanks for the post - your project looks really interesting!
    – user87422
    Sep 21, 2009 at 14:34

3 Answers 3

1

My solution was to put the config in the same database as the application. That way, I could simply pass one DB connector to the app and it would use the correct config.

In the app, the config was accessed via a global config instance which would read all values from the database in one go and cache them. For a web based app, I use a special URL to tell the config instance to refresh itself.

For other apps, I use a local file. When the file exists, the config data is reread. You can do this in a second thread or just check for the file every time. Since the path is static, the OS can optimize this access until it take very little time. After the config has been read, the file is deleted, so I know that the reload happened.

2
  • One of the config elements we need to draw in is what database name to connect to... while putting every deployment into one big DB is possible, it has some clients a bit worried and so having discrete DB's and 'partitioning' them provides a bit of natural sharding as well as a sense of added security/privacy for clients. I guess my other question is whether something like MySQL using InnoDB or MyISAM would be as 'quick' in a read heavy environment as some other database types...
    – user87422
    Sep 21, 2009 at 13:49
  • My suggestion is to give the DB name as the sole config element to the app. So you can have many DBs and each contains a copy of the config for itself. Example: You have DBs "dev", "test", "prod". Same app, each gets the name as startup argument and then reads the correct config from the right DB. Sep 21, 2009 at 13:54
0

disclaimer: I work on a tool to assist deployment (but I'm not advocating it's usage for you)

I suggest you just have a private website that distributes the configs, and simply cache them in your 'reading' apps. Then you may have the ability for the central server to 'ping' the externals, when an update occurs, suggesting that they refresh their cache (or have that happen as part of a cache expiry on the clients).

4
  • Good tip; I'm not sure it would work in this case cause we're using a common code-based in the application servers which could have hits coming in on one domain at one moment, and then another domain (deployment) the next moment.. they kinda can't remember their context/configs unfortunately.
    – user87422
    Sep 21, 2009 at 13:47
  • Geoff: Then you key your cache on that basis.
    – Noon Silk
    Sep 21, 2009 at 14:05
  • So, the idea would be to store all the config variables in memory, and then access them at run-time that way? Can't argue with the 'speed' benefits of this approach... isolating namespace aspects might be a bit of fun.
    – user87422
    Sep 21, 2009 at 14:44
  • Indeed; You can store them where you want; it could be memcached, it could be a file, it could be a DB (as someone else suggests); the main trick is to get them from a central location, and only update from that, based on expiry.
    – Noon Silk
    Sep 21, 2009 at 14:51
0

This may be a "non answer" of sorts, but consider the following: are you trying to solve a perceived problem in a way that would introduce even more issues in the future? You've mentioned your reliance on Perl data structures that live inside config files. Consider for a moment that all this configuration data lives in a RDBMS. How would you go about replicating the functionality that you get "for free" with your existing approach, in a system that provides a very different set of functionality? Wouldn't you at some point end up with CLOB columns containing your good old Perl data structures?

What exactly is driving you away from your current configuration mechanism? I have a feeling that those issues could be addressed on their own without necessitating a large-scale architectural upheaval.

1
  • We're not using too much in the way of structures; we could go back to key/value pairs and through naming them 'slurp' them into the Perl data structures we already have if we need to. We just need to get to a point where we are able to deploy new deployments/domains without touching any .pl or static configs. Load balancing as well as optimizing a core code set are the drivers. Still, your warning about thinking through consequences and testing non-existent concerns is appreciated... best not to go for overkill.
    – user87422
    Sep 21, 2009 at 14:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy