2

The following code produces different results in python 2.7.5.final.0 with pandas 0.15.1 and numpy 1.9.1 and in python 2.7.11.final.0 with pandas 0.18.0 and numpy 1.10.4 (the anaconda package).

The former version gives the result 18292498239.8; the latter, 18292498239.824.

import numpy as np
import pandas as pd
x = 18292498239.824
df = pd.DataFrame({'One': x},index=["bignum"])
df.to_csv('junktest.txt')
fh = open('junktest.txt','rb')
res = fh.read().split('\n')[1].split(',')[1]
print "Result:",res

But if we set x to 292498239.824, we get the same result from both: 292498239.824. If we go up an order of magnitude (x = 118292498239.824), the results are 1.1829249824e+11 and 118292498239.824.

It looks like the later version of pandas.DataFrame.to_csv() restricts floats to 12 digits, but I cannot find anything in the pandas documentation to tell when the change occurred -- or why.

This caused some of my unit tests to fail upon upgrading to anaconda; I would like to be able to upgrade without having to substantially revise my tests.

1 Answer 1

2

UPDATE2:

you can try to use np.set_printoptions(precision=20) function:

np.set_printoptions(precision=20)
df.to_csv('d:/temp/a.csv', index=False)

gives me

d:/temp/a.csv:

One
18292498239.824
123456789012345.12

NOTE: using pandas 0.18.1

DF:

In [25]: df
Out[25]:
            One
0  1.829250e+10
1  1.234568e+14

OLD answer:

use float_format='%16.4f' parameter in the to_csv() call

Docs:

float_format : string, default None

Format string for floating point numbers

5
  • Thanks, but that doesn't work. Not all columns require three decimal places and I don't want trailing zeroes -- unless there's a way to have different float_formats on different columns? I looked, but didn't see any such .... Commented May 4, 2016 at 19:37
  • it looks like astype(str) also has a 12-character limit. I want the first number to be saved to csv as 18292498239.824. This was the former behavior, after all. Commented May 5, 2016 at 16:23
  • 2
    The problem is not that to_csv() changed, but that csv.writer changed. Trying to find when and how this change was made, I found this: bugs.python.org/issue13573. So, in December 2011, csv.writer was changed to use __repr__ instead of __str__ for floats. Then, sometime before February 2016, it was changed back to use __str__ instead of __repr__ and the documentation was changed to reflect this in Feburary 2016. Or, perhaps, the note on bugs.python.org is incorrect .... Commented May 9, 2016 at 18:56
  • @F.Bartlett, just tested np.set_printoptions(precision=20) before calling to_csv() - it does exactly what you need. Tested with pandas 0.18.1 Commented May 9, 2016 at 19:08
  • 1
    I'm using pandas 0.18.0, which may make a difference. But, even after setting precision=20, df = pd.DataFrame({'One': 18292498239.824},index=["bignum"]) df.to_csv('repr_test.csv') !cat repr_test.csv produces ,One bignum,18292498239.8 We want to be able to use to_csv to save a DataFrame and then recreate a precisely identical copy from the csv file (our testing indicated that this was faster than other methods). The use of __str__ instead of __repr__ prevents this; it's not really a matter of precision. Commented May 10, 2016 at 19:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.