126

the following code worked until today when I imported from a Windows machine and got this error:

new-line character seen in unquoted field - do you need to open the file in universal-newline mode?

import csv

class CSV:


    def __init__(self, file=None):
        self.file = file

    def read_file(self):
        data = []
        file_read = csv.reader(self.file)
        for row in file_read:
            data.append(row)
        return data

    def get_row_count(self):
        return len(self.read_file())

    def get_column_count(self):
        new_data = self.read_file()
        return len(new_data[0])

    def get_data(self, rows=1):
        data = self.read_file()

        return data[:rows]

How can I fix this issue?

def upload_configurator(request, id=None):
    """
    A view that allows the user to configurator the uploaded CSV.
    """
    upload = Upload.objects.get(id=id)
    csvobject = CSV(upload.filepath)

    upload.num_records = csvobject.get_row_count()
    upload.num_columns = csvobject.get_column_count()
    upload.save()

    form = ConfiguratorForm()

    row_count = csvobject.get_row_count()
    colum_count = csvobject.get_column_count()
    first_row = csvobject.get_data(rows=1)
    first_two_rows = csvobject.get_data(rows=5)
1

9 Answers 9

185

It'll be good to see the csv file itself, but this might work for you, give it a try, replace:

file_read = csv.reader(self.file)

with:

file_read = csv.reader(self.file, dialect=csv.excel_tab)

Or, open a file with universal newline mode and pass it to csv.reader, like:

reader = csv.reader(open(self.file, 'rU'), dialect=csv.excel_tab)

Or, use splitlines(), like this:

def read_file(self):
    with open(self.file, 'r') as f:
        data = [row for row in csv.reader(f.read().splitlines())]
    return data
7
  • This now gives the same error, but on line starting upload.num_records = csvobject.get_row_count() now
    – GrantU
    Commented Jun 26, 2013 at 9:19
  • 1
    and when I try the split lines version (which is very cools thanks) I get coercing to Unicode: need string or buffer, S3BotoStorageFile found
    – GrantU
    Commented Jun 26, 2013 at 9:21
  • 4
    What option eventually worked? Btw, you are reading the file twice: in get_row_count() and in get_column_count() - consider reading the file in __init__ and remember data in self.data, then use it in other methods.
    – alecxe
    Commented Jun 26, 2013 at 9:47
  • 1
    +1 for splitlines() which avoids messing around with different formatting options on OSX. Hope it works across other platforms too... Commented Aug 2, 2017 at 4:36
  • Great answer. Using - "dialect=csv.excel_tab" however, screws up the output when used with csv.DictReader. Just the 'rU' options works magically though
    – Murphy
    Commented Oct 19, 2017 at 0:06
55

I realize this is an old post, but I ran into the same problem and don't see the correct answer so I will give it a try

Python Error:

_csv.Error: new-line character seen in unquoted field

Caused by trying to read Macintosh (pre OS X formatted) CSV files. These are text files that use CR for end of line. If using MS Office make sure you select either plain CSV format or CSV (MS-DOS). Do not use CSV (Macintosh) as save-as type.

My preferred EOL version would be LF (Unix/Linux/Apple), but I don't think MS Office provides the option to save in this format.

4
  • 4
    MS DOS Comma Separated didn't work for me (same error), but Windows Comma Separated.
    – tmthyjames
    Commented May 21, 2015 at 15:48
  • 3
    If you are on a Mac, this is absolutely the correct answer.
    – HashHazard
    Commented Jun 30, 2015 at 17:48
  • I get the same issue on OS X. I find myself having to make a new CSV file. Simply saving the current as plain CSV format or CSV (MS-DOS) does not fix the issue.
    – Pyderman
    Commented Feb 13, 2016 at 14:48
  • 1
    On OS X, Windows Comma Separated csv worked, MS DOS Comma Separated didn't. Commented Feb 9, 2017 at 2:14
33

For Mac OS X, save your CSV file in "Windows Comma Separated (.csv)" format.

1
  • 1
    thanks, that was the needed ingredient, as I'm using Mac w/ MS office. Commented Feb 24, 2016 at 19:50
19

If this happens to you on mac (as it did to me):

  1. Save the file as CSV (MS-DOS Comma-Separated)
  2. Run the following script

    with open(csv_filename, 'rU') as csvfile:
        csvreader = csv.reader(csvfile)
        for row in csvreader:
            print ', '.join(row)
    
0
5

Try to run dos2unix on your windows imported files first

2
  • no really an option I need to allow user to upload csv from both Windows and Macs without any special modification. The import was saved from Excel (Windows) as a CSV so maybe there is something extra that needs to be done in Python to read these?
    – GrantU
    Commented Jun 26, 2013 at 9:02
  • @GrantU You are referring to Mac OS X 10.0 or later, not Mac OS 9 or earlier, correct? Between 9 and 10, Mac OS switched from \x0d (ProDOS) line endings to \x0a (UNIX) line endings. Commented Nov 28, 2016 at 18:03
2

This is an error that I faced. I had saved .csv file in MAC OSX.

While saving, save it as "Windows Comma Separated Values (.csv)" which resolved the issue.

1

This worked for me on OSX.

# allow variable to opened as files
from io import StringIO

# library to map other strange (accented) characters back into UTF-8
from unidecode import unidecode

# cleanse input file with Windows formating to plain UTF-8 string
with open(filename, 'rb') as fID:
    uncleansedBytes = fID.read()
    # decode the file using the correct encoding scheme
    # (probably this old windows one) 
    uncleansedText = uncleansedBytes.decode('Windows-1252')

    # replace carriage-returns with new-lines
    cleansedText = uncleansedText.replace('\r', '\n')

    # map any other non UTF-8 characters into UTF-8
    asciiText = unidecode(cleansedText)

# read each line of the csv file and store as an array of dicts, 
# use first line as field names for each dict. 
reader = csv.DictReader(StringIO(cleansedText))
for line_entry in reader:
    # do something with your read data 
1

I know this has been answered for quite some time but not solve my problem. I am using DictReader and StringIO for my csv reading due to some other complications. I was able to solve problem more simply by replacing delimiters explicitly:

with urllib.request.urlopen(q) as response:
    raw_data = response.read()
    encoding = response.info().get_content_charset('utf8') 
    data = raw_data.decode(encoding)
    if '\r\n' not in data:
        # proably a windows delimited thing...try to update it
        data = data.replace('\r', '\r\n')

Might not be reasonable for enormous CSV files, but worked well for my use case.

1
  • That solved my problem,Thanks!Look here
    – aof
    Commented Feb 23, 2020 at 13:11
0

Alternative and fast solution : I faced the same error. I reopened the "wierd" csv file in GNUMERIC on my lubuntu machine and exported the file as csv file. This corrected the issue.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.