102

I have a Python application which needs quite a few (~30) configuration parameters. Up to now, I used the OptionParser class to define default values in the app itself, with the possibility to change individual parameters at the command line when invoking the application.

Now I would like to use 'proper' configuration files, for example from the ConfigParser class. At the same time, users should still be able to change individual parameters at the command line.

I was wondering if there is any way to combine the two steps, e.g. use optparse (or the newer argparse) to handle command line options, but reading the default values from a config file in ConfigParse syntax.

Any ideas how to do this in an easy way? I don't really fancy manually invoking ConfigParse, and then manually setting all defaults of all the options to the appropriate values...

2

11 Answers 11

110

I just discovered you can do this with argparse.ArgumentParser.parse_known_args(). Start by using parse_known_args() to parse a configuration file from the commandline, then read it with ConfigParser and set the defaults, and then parse the rest of the options with parse_args(). This will allow you to have a default value, override that with a configuration file and then override that with a commandline option. E.g.:

Default with no user input:

$ ./argparse-partial.py
Option is "default"

Default from configuration file:

$ cat argparse-partial.config 
[Defaults]
option=Hello world!
$ ./argparse-partial.py -c argparse-partial.config 
Option is "Hello world!"

Default from configuration file, overridden by commandline:

$ ./argparse-partial.py -c argparse-partial.config --option override
Option is "override"

argprase-partial.py follows. It is slightly complicated to handle -h for help properly.

import argparse
import ConfigParser
import sys

def main(argv=None):
    # Do argv default this way, as doing it in the functional
    # declaration sets it at compile time.
    if argv is None:
        argv = sys.argv

    # Parse any conf_file specification
    # We make this parser with add_help=False so that
    # it doesn't parse -h and print help.
    conf_parser = argparse.ArgumentParser(
        description=__doc__, # printed with -h/--help
        # Don't mess with format of description
        formatter_class=argparse.RawDescriptionHelpFormatter,
        # Turn off help, so we print all options in response to -h
        add_help=False
        )
    conf_parser.add_argument("-c", "--conf_file",
                        help="Specify config file", metavar="FILE")
    args, remaining_argv = conf_parser.parse_known_args()

    defaults = { "option":"default" }

    if args.conf_file:
        config = ConfigParser.SafeConfigParser()
        config.read([args.conf_file])
        defaults.update(dict(config.items("Defaults")))

    # Parse rest of arguments
    # Don't suppress add_help here so it will handle -h
    parser = argparse.ArgumentParser(
        # Inherit options from config_parser
        parents=[conf_parser]
        )
    parser.set_defaults(**defaults)
    parser.add_argument("--option")
    args = parser.parse_args(remaining_argv)
    print "Option is \"{}\"".format(args.option)
    return(0)

if __name__ == "__main__":
    sys.exit(main())
10
  • 27
    I've been asked above reusing the above code and I hereby place it into the pubic domain.
    – Von
    Oct 9, 2013 at 1:20
  • 25
    'pubic domain' made me laugh. I am just a stupid kid.
    – SylvainD
    Jun 19, 2014 at 13:50
  • 2
    argh! this is really cool code, but SafeConfigParser interpolating of properties overriden by command line doesn't work. E.g. if you add the following line to argparse-partial.config another=%(option)s you are cruel then another would always resolve to Hello world you are cruel even if option is overriden to something else in command line.. argghh-parser!
    – ihadanny
    Jul 29, 2014 at 10:30
  • 2
    Note that set_defaults only works if the argument names don't contain dashes or underscores. So one can opt for --myVar instead of --my-var (which is, unfortunately, quite ugly). To enable case-sensitivity for the config file, use config.optionxform = str before parsing the file, so myVar doesn't get transformed to myvar. Mar 11, 2016 at 11:48
  • 1
    Note that if you want to add --version option to your application, it's better to add it to conf_parser than to parser and exit application after printing help. If you add --version to parser and you start application with --version flag, than your application needlessly try to open and parse args.conf_file configuration file (which can be malformed or even non-existent, which leads to exception). Apr 19, 2017 at 10:50
27

Check out ConfigArgParse - its a new PyPI package (open source) that serves as a drop in replacement for argparse with added support for config files and environment variables.

2
  • 3
    just tried it and wit works great :) Thanks for pointing this out.
    – red_tiger
    Apr 22, 2016 at 13:05
  • 3
    Thanks - looks good! That web page also compares ConfigArgParse with other options, including argparse, ConfArgParse, appsettings, argparse_cnfig, yconf, hieropt, and configurati
    – nealmcb
    Jan 20, 2017 at 2:18
9

I'm using ConfigParser and argparse with subcommands to handle such tasks. The important line in the code below is:

subp.set_defaults(**dict(conffile.items(subn)))

This will set the defaults of the subcommand (from argparse) to the values in the section of the config file.

A more complete example is below:

####### content of example.cfg:
# [sub1]
# verbosity=10
# gggg=3.5
# [sub2]
# host=localhost

import ConfigParser
import argparse

parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()

parser_sub1 = subparsers.add_parser('sub1')
parser_sub1.add_argument('-V','--verbosity', type=int, dest='verbosity')
parser_sub1.add_argument('-G', type=float, dest='gggg')

parser_sub2 = subparsers.add_parser('sub2')
parser_sub2.add_argument('-H','--host', dest='host')

conffile = ConfigParser.SafeConfigParser()
conffile.read('example.cfg')

for subp, subn in ((parser_sub1, "sub1"), (parser_sub2, "sub2")):
    subp.set_defaults(**dict(conffile.items(subn)))

print parser.parse_args(['sub1',])
# Namespace(gggg=3.5, verbosity=10)
print parser.parse_args(['sub1', '-V', '20'])
# Namespace(gggg=3.5, verbosity=20)
print parser.parse_args(['sub1', '-V', '20', '-G','42'])
# Namespace(gggg=42.0, verbosity=20)
print parser.parse_args(['sub2', '-H', 'www.example.com'])
# Namespace(host='www.example.com')
print parser.parse_args(['sub2',])
# Namespace(host='localhost')
1
  • my problem is that argparse sets the config file path, and the config file sets argparse defaults... stupid chicken-egg problem
    – olivervbk
    Jan 15, 2011 at 1:09
5

I can't say it's the best way, but I have an OptionParser class that I made that does just that - acts like optparse.OptionParser with defaults coming from a config file section. You can have it...

class OptionParser(optparse.OptionParser):
    def __init__(self, **kwargs):
        import sys
        import os
        config_file = kwargs.pop('config_file',
                                 os.path.splitext(os.path.basename(sys.argv[0]))[0] + '.config')
        self.config_section = kwargs.pop('config_section', 'OPTIONS')

        self.configParser = ConfigParser()
        self.configParser.read(config_file)

        optparse.OptionParser.__init__(self, **kwargs)

    def add_option(self, *args, **kwargs):
        option = optparse.OptionParser.add_option(self, *args, **kwargs)
        name = option.get_opt_string()
        if name.startswith('--'):
            name = name[2:]
            if self.configParser.has_option(self.config_section, name):
                self.set_default(name, self.configParser.get(self.config_section, name))

Feel free to browse the source. Tests are in a sibling directory.

4

Update: This answer still has issues; for example, it cannot handle required arguments, and requires an awkward config syntax. Instead, ConfigArgParse seems to be exactly what this question asks for, and is a transparent, drop-in replacement.

One issue with the current is that it will not error if the arguments in the config file are invalid. Here's a version with a different downside: you'll need to include the -- or - prefix in the keys.

Here's the python code (Gist link with MIT license):

# Filename: main.py
import argparse

import configparser

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument('--config_file', help='config file')
    args, left_argv = parser.parse_known_args()
    if args.config_file:
        with open(args.config_file, 'r') as f:
            config = configparser.SafeConfigParser()
            config.read([args.config_file])

    parser.add_argument('--arg1', help='argument 1')
    parser.add_argument('--arg2', type=int, help='argument 2')

    for k, v in config.items("Defaults"):
        parser.parse_args([str(k), str(v)], args)

    parser.parse_args(left_argv, args)
print(args)

Here's an example of a config file:

# Filename: config_correct.conf
[Defaults]
--arg1=Hello!
--arg2=3

Now, running

> python main.py --config_file config_correct.conf --arg1 override
Namespace(arg1='override', arg2=3, config_file='test_argparse.conf')

However, if our config file has an error:

# config_invalid.conf
--arg1=Hello!
--arg2='not an integer!'

Running the script will produce an error, as desired:

> python main.py --config_file config_invalid.conf --arg1 override
usage: test_argparse_conf.py [-h] [--config_file CONFIG_FILE] [--arg1 ARG1]
                             [--arg2 ARG2]
main.py: error: argument --arg2: invalid int value: 'not an integer!'

The main downside is that this uses parser.parse_args somewhat hackily in order to obtain the error checking from ArgumentParser, but I am not aware of any alternatives to this.

3

You can use ChainMap

A ChainMap groups multiple dicts or other mappings together to create a single, updateable view. If no maps are specified, a single empty dictionary is provided so that a new chain always has at least one mapping.

You can combine values from command line, environment variables, configuration file, and in case if the value is not there define a default value.

import os
from collections import ChainMap, defaultdict

options = ChainMap(command_line_options, os.environ, config_file_options,
               defaultdict(lambda: 'default-value'))
value = options['optname']
value2 = options['other-option']


print(value, value2)
'optvalue', 'default-value'
1
  • What's the advantage of a ChainMap over say a chain of dicts updated in the precedence order desired? With defaultdict there's possibly an advantage as novel or unsupported options can be set but that's separate from ChainMap. I assume I'm missing something.
    – Dan
    Jan 1, 2020 at 1:40
3

fromfile_prefix_chars

Maybe not the cleanest of APIs, but worth knowing about.

main.py

#!/usr/bin/env python3
import argparse
parser = argparse.ArgumentParser(fromfile_prefix_chars='@')
parser.add_argument('-a', default=13)
parser.add_argument('-b', default=42)
print(parser.parse_args())

Then:

$ printf -- '-a\n1\n-b\n2\n' > opts.txt
$ ./main.py
Namespace(a=13, b=42)
$ ./main.py @opts.txt
Namespace(a='1', b='2')
$ ./main.py @opts.txt -a 3 -b 4
Namespace(a='3', b='4')
$ ./main.py -a 3 -b 4 @opts.txt
Namespace(a='1', b='2')

Documentation: https://docs.python.org/3.6/library/argparse.html#fromfile-prefix-chars

This @opts.txt convention has some precedents e.g. in the GCC toolchain: What does "@" at the command line mean?

How to use a proper CLI option to indicate the options file rather than the ugly @ thing: how to get argparse to read arguments from a file with an option rather than prefix

Tested on Python 3.6.5, Ubuntu 18.04.

1

Try to this way

# encoding: utf-8
import imp
import argparse


class LoadConfigAction(argparse._StoreAction):
    NIL = object()

    def __init__(self, option_strings, dest, **kwargs):
        super(self.__class__, self).__init__(option_strings, dest)
        self.help = "Load configuration from file"

    def __call__(self, parser, namespace, values, option_string=None):
        super(LoadConfigAction, self).__call__(parser, namespace, values, option_string)

        config = imp.load_source('config', values)

        for key in (set(map(lambda x: x.dest, parser._actions)) & set(dir(config))):
            setattr(namespace, key, getattr(config, key))

Use it:

parser.add_argument("-C", "--config", action=LoadConfigAction)
parser.add_argument("-H", "--host", dest="host")

And create example config:

# Example config: /etc/myservice.conf
import os
host = os.getenv("HOST_NAME", "localhost")
1
1

Worth mentioning here is jsonargparse, with MIT license and available on PyPI. It is also an extension of argparse that supports loading from config files and environment variables. It is similar to ConfigArgParse, but it is newer, with many more useful features and well maintained.

An example main.py would be:

from jsonargparse import ArgumentParser, ActionConfigFile

parser = ArgumentParser()
parser.add_argument("--config", action=ActionConfigFile)
parser.add_argument("--opt1", default="default 1")
parser.add_argument("--opt2", default="default 2")
args = parser.parse_args()
print(args.opt1, args.opt2)

Having a config file config.yaml with content:

opt1: one
opt2: two

Then an example run from the command line:

$ python main.py --config config.yaml --opt1 ONE
ONE two
0

parse_args() can take an existing Namespace and merge the existing Namespace with args/options it's currently parsing; the options args/options in the "current parsing" take precedence an override anything in the existing Namespace:

foo_parser = argparse.ArgumentParser()
foo_parser.add_argument('--foo')

ConfigNamespace = argparse.Namespace()
setattr(ConfigNamespace, 'foo', 'foo')

args = foo_parser.parse_args([], namespace=ConfigNamespace)
print(args)
# Namespace(foo='foo')

# value `bar` will override value `foo` from ConfigNamespace
args = foo_parser.parse_args(['--foo', 'bar'], namespace=ConfigNamespace)
print(args)
# Namespace(foo='bar')

I've mocked it up for a real config file option. I'm parsing twice, once, as a "pre-parse" to see if the user passed a config-file, and then again for the "final parse" that integrates the optional config-file Namespace.

I have this very simple JSON config file, config.ini:

[DEFAULT]
delimiter = |

and when I run this:

import argparse
import configparser

parser = argparse.ArgumentParser()
parser.add_argument('-c', '--config-file', type=str)
parser.add_argument('-d', '--delimiter', type=str, default=',')

# Parse cmd-line args to see if config-file is specified
pre_args = parser.parse_args()

# Even if config is not specified, need empty Namespace to pass to final `parse_args()`
ConfigNamespace = argparse.Namespace()

if pre_args.config_file:
    config = configparser.ConfigParser()
    config.read(pre_args.config_file)

    for name, val in config['DEFAULT'].items():
        setattr(ConfigNamespace, name, val)


# Parse cmd-line args again, merging with ConfigNamespace, 
# cmd-line args take precedence
args = parser.parse_args(namespace=ConfigNamespace)

print(args)

with various cmd-line settings, I get:

./main.py
Namespace(config_file=None, delimiter=',')

./main.py -c config.ini
Namespace(config_file='config.ini', delimiter='|')

./main.py -c config.ini -d \;
Namespace(config_file='config.ini', delimiter=';')
0

I've found the existing answers to come up short, especially when dealing with subcommands and required arguments.

Here's the solution I ended up with:

# Create a separate parser for the --config-file argument
config_parser = argparse.ArgumentParser(
    add_help=False
)
config_parser.add_argument(
    "--config-file",
    metavar="FILE"
)

# Parse the config file arg, but hold on to all the other args
# (we'll need them later, for the main parser)
args, remaining_argv = config_parser.parse_known_args()
config_data = {}
if args.config_file:
    # config_data = <Process your file however you'd like>

# Use the config to generate new CLI arguments
argv = remaining_argv
for k, v in config_data.items():
    argv.append(f"--{k}")
    argv.append(str(v))

# Setup your main parser
parser = argparse.ArgumentParser(
    # ...
)
# Parse arguments
args = parser.parse_args(argv)

Maybe a little funky, but it works well for my use case.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.