2

CKAN Version if known (or site URL):
Latest

Please describe the expected behavior:
Large files get uploaded to the datastore with the datapusher.

Please describe the actual behavior:
Everything works perfectly when uploading normal sized files.

However, when trying to upload large csv files (750 mb, not SO large but still) with the datapusher to the datastore, I get this error :

_"Job "push_to_datastore (trigger: RunTriggerNow, run = True, next run at: None)" raised an exception
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/apscheduler/scheduler.py", line 512, in _run_job
    retval = job.func(*job.args, **job.kwargs)
  File "/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py", line 364, in push_to_datastore
    f = cStringIO.StringIO(response.read())
  File "/usr/lib/python2.7/socket.py", line 359, in read
    return buf.getvalue()
MemoryError"_

I changed the memory allowed size in the datapusher_settings.py, on my ckan conf file, in my nginx_proxy_cache, but I don't see what's wrong.

More details of what I changed :

ckan.max_resource_size = 5120 in production.ini

MAX_CONTENT_LENGTH = 1024000000 in https://github.com/ckan/datapusher/blob/master/datapusher/jobs.py#L28

The cache size in proxy_cache_path /tmp/nginx_cache levels=1:2 keys_zone=cache:30m max_size=250m; and the client_max_body_size 10000M; in nginx conf file.

I do need help for this. Thanks in advance, Twinko5

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.