1

I'm trying to read a csv into a new dataframe with pandas. A number of the columns may only contain numeric values, but I still want to have them imported in as strings/objects, rather than having columns of float type.

I'm trying to write some python scripts for data conversion/migration. I'm not an advanced Python programmer and mostly learning as I come across a problem that needs solving.

The csvs I am importing have varying number of columns, and even different column titles, and in any order, over which I have no control, so I can't explicitly specify data types using the dtype parameter with read_csv. I just want any column imported to be treated as an object data type so I can analyse it further for data quality.

Examples would be 'Staff ID', and 'License Number' columns on one CSV I tried which should be strings fields holding 7-digit IDs, being imported as type float64.

I have tried using astype with read_csv and apply map on the dataframe after import

Note, there is no hard and fast rule on the contents of the type or quality of the data which is why I want to always import them as dtype of object.

Thanks in advance for anyone who can help me figure this out.

I've used the following code to read it in.

import pandas as pd
df = pd.read_csv("agent.csv",encoding="ISO-8859-1")

This creates the 'License Number' column in df with a type of float64 (among others).

Here is an example of License Number which should be a string:

'1275595' being stored as 1275595.0

Converting it back to a string/object in df after the import changes it back to '1275595.0'

5
  • first you should read documentation for read_csv. There should be option to define convert for every column separatelly.
    – furas
    Jul 4, 2019 at 0:48
  • my understanding from the documentation is that I need to know the number of columns in order to define the conversion for each column separately. There could be 2 columns or 20. I've come up with some as much as 70 columns.
    – Peter
    Jul 4, 2019 at 1:00
  • try pd.read_csv(..., dtype=str). It should stop converting data.
    – furas
    Jul 4, 2019 at 1:29
  • Cool. Thanks. Seems to have done the trick.
    – Peter
    Jul 4, 2019 at 1:46

2 Answers 2

2

It should stop converting data.

pd.read_csv(..., dtype=str)

Doc: read_csv

dtype: ...  Use str or object together with suitable na_values settings 
            to preserve and not interpret dtype. 
0

I recommend you split your csv reading process into multiple, specific-purpose functions.

For example:

import pandas as pd

# Base function for reading a csv. All the parsing/formatting is done here
def read_csv(file_content, header=False, columns=None, encoding='utf-8'):
    df = pd.read_csv(file_content, header=header, encoding=encoding)
    df.columns = columns
    return df

# Function with a specific purpose as stated in the name.
def read_csv_license_plates(file_content, encoding='utf-8'):
    columns = ['col1', 'col2', 'col3']
    df = read_csv(file_content, True, columns)
    return df

read_csv_license_plates('agent.csv', encoding='ISO-8859-1')

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.