I am trying to read a file containing data for different dates using numpy.genfromtxt()
in python3. The file basically looks like
Date,Open,High,Low,Close,Volume
1-Apr-15,108.33,108.66,108.33,108.66,290
but may contain missing values marked as -
.
The following code works fine in python2
str2date = lambda x: datetime.strptime(x, '%d-%b-%y').strftime('%Y-%m-%d')
data = np.genfromtxt('test.dat', dtype="S9,f8,f8,f8,f8,f8", delimiter=',', names=True, missing_values='-', converters={0: str2date})
but fails in python3 with
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)
locale.getpreferredencoding(False)
returns UTF-8 as the default encoding and the suggested solution by setting the encoding for the input stream suggested for example here is a bit tricky. I also tried setting the encoding of the terminal without success. I also have to admit, that I do not see a solution to my problem in this answer as there are no special characters contained in the file -- or at least I do not see them.
How can I solve this issue without stepping back to python2?
genfromtxt(open('test.dat', encoding='utf-8'))
complains about bytes provides instead of a string. But pandas works like a charm. Thanks :). It you put that in an answer I'll accept it.genfromtxt
opens the file in binary mode, and works with bytestrings (Py3). Theconverters
solution in stackoverflow.com/questions/33001373/… doesn't help?