4

I have a csv column with integers that has nulls in it, that I want to read with pandas. In the sample file below the column nr_ok has the same content that the column nr_nan, except for the missing value in the first row

row,nr_ok,nr_nan
1,4696374908103381,
2,1780963748798374342,1780963748798374342
3,719826117241460269,719826117241460269

I'm working with pandas version 1.3.4. If I attempt to read it without dtype specification, pandas converts the second column to float and I lose precision. So I tried specifiying dtype, using the Nullable type 'Int64'

type_dict = {'nr_nan':'Int64'}
df2 = pd.read_csv('test_int64.csv', dtype = type_dict)
df2

As a result I still lose precision. Notice how the last digits of the numbers have changed compared to the original

  | row |               nr_ok |              nr_nan
------------------------------------------------------  
0 |   1 |    4696374908103381 |                <NA>
1 |   2 | 1780963748798374342 | 1780963748798374400
2 |   3 |  719826117241460269 |  719826117241460224

It seems to me that pandas is reading the column as float and converting it to integer after that, with the subsequent precision loss.
Is there any way I can read a file with integer columns containing null values without losing precision?

1 Answer 1

2

Try using:

df2 = pd.read_csv('test_int64.csv', converters={'nr_nan':lambda x: pd.NA if x == '' else int(x)})
df2.nr_nan = df2.nr_nan.astype('Int64')

The first line will use a custom converter and will result in a mixed type field (a mix of 64 bit integers and pd.NA) The second line converts to the nullable int type

2
  • 1
    whythis isn't a bug in pandas? Commented Sep 20, 2022 at 9:45
  • I agree - I haven't had time to check if an issue has already been raised.
    – Mike
    Commented Sep 20, 2022 at 19:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.