What's the latest and greatest for fast YAML parsing in Python? Syck is out of date and recommends using PyYaml, yet PyYaml is pretty slow, and suffers from the GIL problem:

>>> def xit(f, x):
        import threading
        for i in xrange(x):
                threading.Thread(target=f).start()

>>> def stressit():
        start = time.time()
        res = yaml.load(open(path_to_11000_byte_yaml_file))
        print "Took %.2fs" % (time.time() - start,)    

>>> xit(stressit, 1)
Took 0.37s
>>> xit(stressit, 2)
Took 1.40s
Took 1.41s
>>> xit(stressit, 4)
Took 2.98s
Took 2.98s
Took 2.99s
Took 3.00s

Given my use case I can cache the parsed objects, but I'd still prefer a faster solution even for that.

share|improve this question
    
Hmm I had no idea YAML was so relatively under-used and not really active much... might just use JSON next time as all I really need is to encode lists, numbers, strings, and dicts.. I kind of prefer how YAML looks but ah well – Claudiu May 31 '13 at 19:14
1  
PyYAML has both a pure Python implementation and a wrapper around a C library. Before you benchmark, make sure the C library is available so PyYAML can use it. – delnan May 31 '13 at 20:33
    
@delnan: Ah I wasn't aware, as the site doesn't make that obvious. I will try that & check back. – Claudiu Jun 1 '13 at 13:10
up vote 6 down vote accepted

The linked wiki page states after the warning "Use libyaml (c), and PyYaml (python)". Although the note does have a bad wikilink (should be PyYAML not PyYaml).

As for performance, depending on how you installed PyYAML you should have the CParser class available which implements a YAML parser written in optimized C. While I don't think this gets around the GIL issue, it is markedly faster. Here are a few cursory benchmarks I ran on my machine (AMD Athlon II X4 640, 3.0GHz, 8GB RAM):

First with the default pure-Python parser:

$ /usr/bin/python2 -m timeit -s 'import yaml; y=file("large.yaml", "r").read()' \
    'yaml.load(y)'                    
10 loops, best of 3: 405 msec per loop

With the CParser:

$ /usr/bin/python2 -m timeit -s 'import yaml; y=file("large.yaml", "r").read()' \
    'yaml.load(y, Loader=yaml.CLoader)'
10 loops, best of 3: 59.2 msec per loop

And, for comparison, with PyPy using the pure-Python parser.

$ pypy -m timeit -s 'import yaml; y=file("large.yaml", "r").read()' \
    'yaml.load(y)'
10 loops, best of 3: 101 msec per loop

For large.yaml I just googled for "large yaml file" and came across this:

https://gist.github.com/nrh/667383/raw/1b3ba75c939f2886f63291528df89418621548fd/large.yaml

(I had to remove the first couple of lines to make it a single-doc YAML file otherwise yaml.load complains.)

EDIT:

Another thing to consider is using the multiprocessing module instead of threads. This gets around GIL problems, but does require a bit more boiler-plate code to communicate between the processes. There are a number of good libraries available though to make multiprocessing easier. There's a pretty good list of them here.

share|improve this answer
    
You seem to really be confused about PIL and GIL. PIL is the Python Imaging Library. Pretty much the go-to library for almost any kind of image manipulation in Python. GIL is the Global Interpreter Lock, which is an implementation detail of the standard python.org Python (CPython) but not other implementations such as Jython or IronPython. – John Y Aug 2 '13 at 21:38
1  
Lol, I know the difference, must just be my dyslexia acting up. Thanks for the corrections though. :) – Isaac Freeman Aug 4 '13 at 23:31

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.