25

I am using the pandas library to write the contents of a mysql database to a csv file.

But when I write the CSV, every other line is blank:

blank lines in csv

Also, it's printing line numbers to the left that I do not want. The first column should be 'Account Number'.

Here is my code:

destination = 'output_file.txt'
read_sql = """ SELECT LinkedAccountId,ProductName,ItemDescription,ResourceId,UnBlendedCost,UnBlendedRate,Name,Owner,Engagement FROM billing_info ;"""
fieldnames = ['Account Number', 'Product Name', 'Item Description', 'Resource ID', 'UnBlended Cost', 'UnBlended Rate', 'Name', 'Owner', 'Engagement']
# Open the file
f = open(destination, 'w')
cursor.execute(read_sql)
while True:
    # Read the data
    df = pd.DataFrame(cursor.fetchmany(1000))
    # We are done if there are no data
    if len(df) == 0:
        break
    # Let's write to the file
    else:
        df.to_csv(f, header=fieldnames)

Why is it printing blank lines between the lines with data? How can I get it to create the file without blank lines and without the line number column to the left?

2
  • 1
    try index=False to exclude the row identifiers. May 31, 2019 at 15:58
  • 3
    What operating system are you running in? Pandas to_csv() in MS Windows seems to set EOL as \r\r\n giving extra blank rows Jul 1, 2020 at 22:29

2 Answers 2

33

Have a look at the official documentation for pandas.DataFrame.to_csv

For convenience, I have posted some items of interest here:

lineterminator : string, optional

The newline character or character sequence to use in the output file. Defaults to os.linesep, which depends on the OS in which this method is called (’\n’ for linux, ‘\r\n’ for Windows, i.e.). ⚠️ In older version of pandas, this parameter is called line_terminator.

index : bool, default True

Write row names (index).

Are probably what you're looking for. As for the empty lines, try explicitly specifying a single newline:

df.to_csv(f, header=fieldnames, index=False, lineterminator='\n')
6
  • Isn't this going to wewrite the headers every 1000 rows? You probably want to set header to false after the first iteration.
    – Dan
    May 31, 2019 at 16:07
  • @Dan, Ah correct, I was just copying his call for that bit. Since that isn't in his question, I will probably leave as-is
    – AlgoRythm
    May 31, 2019 at 16:09
  • Thank you! I have partial progress to report. The column showing line numbers at the left are gone! However there are still blank lines in between the lines of data! Any other ideas?
    – bluethundr
    May 31, 2019 at 16:35
  • oh good point about the headers! I'll leave them out of this line. Thanks!
    – bluethundr
    May 31, 2019 at 16:36
  • @bluethundr Try setting line_terminator to something like a semicolon, or maybe even set it to a blank string, otherwise, I'm not sure why there are extra lines. Try opening the CSV file with a plain text editor to see if there are actually extra lines or if it's just the program you are using to view the file
    – AlgoRythm
    Jun 1, 2019 at 20:25
4

I came here just for the title and not removal of index numbers. That is why, for completeness sake, I want to add to the accepted answer, that removing the double linebreaks is done just by line_terminator='\n'.

In this example this would be

f = open(destination, 'w')
df.to_csv(f, line_terminator='\n')
f.close()

or when using 'with open(..)'

with open(destination, 'w') as f
    f.write(df.to_csv(line_terminator='\n'))

Other options such as headers can be added to df.to_csv() as needed.

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.