3

I'm trying to bulk load around 200M lines (3.5GB) of data to an Amazon RDS postgresql DB using the following command:

cat data.csv | psql -h<host>.rds.amazonaws.com -U<user> <db> -c "COPY table FROM STDIN  DELIMITER AS ','"

After a couple of minutes I get this error:

connection not open
connection to server was lost

If I run head -n 100000000 data.csv to send the first 100M lines instead of all 200M then the command succeeds instead. I'm guessing that there's a timeout somewhere that's causing the query with the full dataset to fail. I've not been able to find any timeout settings or parameters though.

How can I make the bulk insert succeed with the full dataset?

5
  • Tried any of these methods? stackoverflow.com/questions/3602976/… Dec 2, 2014 at 13:18
  • That's almost exactly what I am doing, and it's timing out for the full dataset Dec 2, 2014 at 17:40
  • 1
    It's similar, but not precisely the same. As I read the statement you're using, it basically creates a giant string, then connects to SQL and then it tries to feed the entire string as argument. If you load psql and run something like \copy ... from '/path/to/data.csv' ..., I'd imagine the connection might stay alive while the file's content is streamed chunk by chunk. Not committing this as answer, since it's a wild guess, but that would be my hunch as to why 10M lines works (= argument pushed entirely before the connection times out) but not the entire file (= argument still uploading). Dec 2, 2014 at 18:10
  • Oh I overlooked that. You're absolutely right. Docs here postgresql.org/docs/9.2/static/…. That did solve the problem! Want to write it up as an answer and get the points? Dec 3, 2014 at 5:23
  • Posted as an answer. Dec 3, 2014 at 10:42

1 Answer 1

3
+200

As I read the statement you're using, it basically creates a giant string, then connects to SQL and then it tries to feed the entire string as argument.

If you load psql and run something like \copy ... from '/path/to/data.csv' ..., I'd imagine the connection might stay alive while the file's content is streamed chunk by chunk.

That would be my hunch as to why 10M lines works (= argument pushed entirely before the connection times out) but not the entire file (= argument still uploading).

1
  • Yeah, using the \copy psql command instead of the COPY statement fixed the timeout issue. Thanks! Dec 3, 2014 at 19:41

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.