I have a problem with filetypes when converting a parquet file to a dataframe.
I do
bucket = 's3://some_bucket/test/usages'
import pyarrow.parquet as pq
import s3fs
s3 = s3fs.S3FileSystem()
read_pq = pq.ParquetDataset(bucket, filesystem=s3).read_pandas()
When I do read_pq
, I get
pyarrow.Table
_COL_0: decimal(9, 0)
_COL_1: decimal(9, 0)
_COL_2: decimal(9, 0)
_COL_3: decimal(9, 0)
When I do df = read_pd.to_pandas(); df.dtypes
, I get
_COL_0 object
_COL_1 object
_COL_2 object
_COL_3 object
dtype: object
The original data are all integers. When I operate on the objects in the pandas dataframe, the operations are very slow.
- How can I convert the parquet columns to a format that will be read as an int or as a float in pandas?
- Or is it best to operate on the pandas dataframe as above and use
pd.to_numeric
or similar? - Or is there an issue with the original dataformat
decimal(9, 0)
?
Or is it best to convert on the pandas dataframe directly?
I tried: read_pq.column('_COL_0').cast('int32')
throws an error like
No cast implemented from decimal(9, 0) to int32