Standing on the shoulders of @ChrisJohnson, I extended the answer from Feb 2016 with a custom dialect for reading. This shell pipeline tool does not need to connect to your database, handles random commas and quotes in the input, and works nicely in Python2 and Python3!
#!/usr/bin/env python
import csv
import sys
# fields are separated by tabs; double-quotes may occur anywhere
csv.register_dialect("mysql", delimiter="\t", quoting=csv.QUOTE_NONE)
tab_in = csv.reader(sys.stdin, dialect="mysql")
comma_out = csv.writer(sys.stdout, dialect=csv.excel)
for row in tab_in:
# print("row: {}".format(row))
comma_out.writerow(row)
Use that print statement to convince yourself it's parsing your input correctly :)
A major caveat: treatment of carriage return characters, ^M aka control-M, \r in linux terms. Altho batch-mode Mysql output correctly escapes embedded newline characters, so there is truly one row per line (defined by linux newline character \n), mysql puts no quotes around column data. If a data item has an embedded carriage-return character, csv.reader rejects that input with this exception:
new-line character seen in unquoted field -
do you need to open the file in universal-newline mode?
Please don't @ me saying I should use universal file mode by re-opening sys.stdin.fileno with mode 'rU'. I tried that, it causes the embedded \r characters to be treated as end-of-record markers, so a single input record is incorrectly transformed into many incomplete output records. I have not found a Python solution to this limitation of Python's csv.reader module. I think the root cause is the csv.reader implementation/limitation noted in their documentation https://docs.python.org/3/library/csv.html#csv.reader:
The reader is hard-coded to recognise either '\r' or '\n' as end-of-line,
and ignores lineterminator.
The weak & unsatisfying solution I can offer is to change each \r character to the two-character sequence '\n' before Python's csv.reader sees the data. I used the sed command. Here's an example of a pipeline with a mysql select and the python script from above:
mysql -u user db --execute="select * from table where id=12345" \
| sed -e 's/\r/\\n/g' \
| mysqlTsvToCsv.py
After fighting this for some time I think Python is not the right solution. If you can live with perl, I think the one-liner script offered by @artfulrobot may be the most-effective and simplest solution.
REPLACE()
in your query to have the quotes escaped. – dsm Dec 10 '08 at 16:08