2

I have the following dataframe and want to pivot it in a way that the column Imprv_Attribute is converted into single columns per key and the values should be Imprv_Attr_Desc. I need the Imprv_Attr_Units information as well, for each newly created column, for example the Imprv_Attr_Units of Bathrooms should get their own column called Bathrooms_Imprv_Attr_Units.

|     | Parcel        | Imprv_Attribute | Imprv_Attr_Desc   | Imprv_Attr_Units |
| --- | ------------- | --------------- | ----------------- | ---------------- |
| 0   | 00002-000-000 | Bathrooms       | 2.0-Baths         | 1.0              |
| 1   | 00002-000-000 | Bedrooms        | 2-2 BEDROOMS      | 1.0              |
| 2   | 00002-000-000 | Exterior Wall   | 13-PRE-FAB PANEL  | 100.0            |
| 3   | 00002-000-000 | Floor Cov       | 08-SHEET VINYL    | 20.0             |
| 4   | 00002-000-000 | Floor Cov       | 14-CARPET         | 80.0             |
| 5   | 00011-000-000 | Bathrooms       | 3.0-Baths         | 1.0              |
| 6   | 00011-000-000 | Bedrooms        | 3-3 BEDROOMS      | 1.0              |
| 7   | 00011-000-000 | Exterior Wall   | 15-CONCRETE BLOCK | 60.0             |
| 8   | 00011-000-000 | Exterior Wall   | 20-FACE BRICK     | 40.0             |
| 9   | 00011-000-000 | Floor Cov       | 14-CARPET         | 100.0            |

My final result should look like this:

| Parcel        | Bathrooms | Bathrooms_Imprv_Attr_Units | Bedrooms     | Bedrooms_Imprv_Attr_Units | Exterior Wall     | Exterior Wall_Imprv_Attr_Units | Floor Cov      | Floor Cov_Imprv_Attr_Unit |
| ------------- | --------- | -------------------------- | ------------ | ------------------------- | ----------------- | ------------------------------ | -------------- | ------------------------- |
| 00002-000-000 | 2.0-Baths | 1.0                        | 2-2 BEDROOMS | 1.0                       | 13-PRE-FAB PANEL  | 100.0                          | 08-SHEET VINYL | 20.0                      |
| 00002-000-000 |           |                            |              |                           |                   |                                | 14-CARPET      | 80.0                      |
| 00011-000-000 | 3.0-Baths | 1.0                        | 3-3 BEDROOMS | 1.0                       | 15-CONCRETE BLOCK | 60.0                           | 14-CARPET      | 100.0                     |
| 00011-000-000 |           |                            |              |                           | 20-FACE BRICK     | 40.0                           |                |                           |

So far I have tried this:

from io import StringIO
import pandas as pd

data = StringIO(
    """
Parcel;Imprv_Attribute;Imprv_Attr_Desc;Imprv_Attr_Units
00002-000-000;Bathrooms;2.0-Baths;1.0
00002-000-000;Bedrooms; 2-2 BEDROOMS;1.0
00002-000-000;Exterior Wall;13-PRE-FAB PANEL;100.0
00002-000-000;Floor Cov;08-SHEET VINYL;   20.0
00002-000-000;Floor Cov;14-CARPET;80.0
00011-000-000;Bathrooms;3.0-Baths;1.0
00011-000-000;Bedrooms; 3-3 BEDROOMS;1.0
00011-000-000;Exterior Wall;15-CONCRETE BLOCK;60.0
00011-000-000;Exterior Wall;20-FACE BRICK;40.0
00011-000-000;Floor Cov;14-CARPET;100.0
"""
)
df = pd.read_csv(data, sep=";")
df = df.pivot_table(values="Imprv_Attr_Desc", index="Parcel", columns="Imprv_Attribute", aggfunc="first")
print(df)

Which results in this dataframe, where I lose the information about Floor Cov and Exterior Wall due to the aggregation function first.

| Parcel        | Bathrooms | Bedrooms     | Exterior Wall     | Floor Cov      |
| ------------- | --------- | ------------ | ----------------- | -------------- |
| 00002-000-000 | 2.0-Baths | 2-2 BEDROOMS | 13-PRE-FAB PANEL  | 08-SHEET VINYL |
| 00011-000-000 | 3.0-Baths | 3-3 BEDROOMS | 15-CONCRETE BLOCK | 14-CARPET      |

I have also tried this answer

df = df.pivot_table(index=[df.index, "Parcel"], columns="Imprv_Attribute", values="Imprv_Attr_Desc")
print(df)

Which results in pandas.core.base.DataError: No numeric types to aggregate. I have also tried a groupby, but this also does not get anywhere near the result I would like:

df_group = df.groupby(["Parcel"])
for key, item in df_group:
    df = df_group.get_group(key)
    df = df.pivot(columns="Imprv_Attribute", values="Imprv_Attr_Desc")
    print(df, "\n\n")
<class 'pandas.core.frame.DataFrame'>
Imprv_Attribute  Bathrooms      Bedrooms     Exterior Wall       Floor Cov           HC&V     HVAC  Heat System Interior Wall  Num Res Units     Roof Type     Roofing
0                2.0-Baths           NaN               NaN             NaN            NaN      NaN          NaN           NaN            NaN           NaN         NaN
1                      NaN  2-2 BEDROOMS               NaN             NaN            NaN      NaN          NaN           NaN            NaN           NaN         NaN
2                      NaN           NaN  13-PRE-FAB PANEL             NaN            NaN      NaN          NaN           NaN            NaN           NaN         NaN
3                      NaN           NaN               NaN  08-SHEET VINYL            NaN      NaN          NaN           NaN            NaN           NaN         NaN
4                      NaN           NaN               NaN       14-CARPET            NaN      NaN          NaN           NaN            NaN           NaN         NaN
5                      NaN           NaN               NaN             NaN  04-FORCED AIR      NaN          NaN           NaN            NaN           NaN         NaN
6                      NaN           NaN               NaN             NaN            NaN      NaN  04-ELECTRIC           NaN            NaN           NaN         NaN
7                      NaN           NaN               NaN             NaN            NaN  01-NONE          NaN           NaN            NaN           NaN         NaN
8                      NaN           NaN               NaN             NaN            NaN      NaN          NaN      04-PANEL            NaN           NaN         NaN
9                      NaN           NaN               NaN             NaN            NaN      NaN          NaN           NaN  Num Res Units           NaN         NaN
10                     NaN           NaN               NaN             NaN            NaN      NaN          NaN           NaN            NaN  03-GABLE/HIP         NaN
11                     NaN           NaN               NaN             NaN            NaN      NaN          NaN           NaN            NaN           NaN  03-ASPHALT

<class 'pandas.core.frame.DataFrame'>
Imprv_Attribute  Bathrooms      Bedrooms      Exterior Wall  Floor Cov           HC&V        HVAC  Heat System Interior Wall  Num Res Units     Roof Type     Roofing
12               3.0-Baths           NaN                NaN        NaN            NaN         NaN          NaN           NaN            NaN           NaN         NaN
13                     NaN  3-3 BEDROOMS                NaN        NaN            NaN         NaN          NaN           NaN            NaN           NaN         NaN
14                     NaN           NaN  15-CONCRETE BLOCK        NaN            NaN         NaN          NaN           NaN            NaN           NaN         NaN
15                     NaN           NaN      20-FACE BRICK        NaN            NaN         NaN          NaN           NaN            NaN           NaN         NaN
16                     NaN           NaN                NaN  14-CARPET            NaN         NaN          NaN           NaN            NaN           NaN         NaN
17                     NaN           NaN                NaN        NaN  04-FORCED AIR         NaN          NaN           NaN            NaN           NaN         NaN
18                     NaN           NaN                NaN        NaN            NaN         NaN  04-ELECTRIC           NaN            NaN           NaN         NaN
19                     NaN           NaN                NaN        NaN            NaN  03-CENTRAL          NaN           NaN            NaN           NaN         NaN
20                     NaN           NaN                NaN        NaN            NaN         NaN          NaN    05-DRYWALL            NaN           NaN         NaN

According to this answer the solution might be a combination of pd.DataFrame.groupby and pd.DataFrame.unstack, but at the moment I don't know how I can apply these in my case.

If anyone has a good idea on how to help me I would greatly appreciate it.

1 Answer 1

1

This can be done with pivot_table (similar to question 10 in the how to pivot canonical), and a few extra steps.

First you need to create a cumcount level so duplicated 'Imprv_Attribute' values within each 'Parcel' get their own label you can use to specify the index with. Then aggregate the multiple value columns (with first). We'll be left with a MultiIndex on the columns that we collapse by imposing your naming convention in a simple list comprehension. Finally, we can sort the columns and remove the cumcount level of the index we created.

df['N'] = df.groupby(['Parcel', 'Imprv_Attribute']).cumcount()

df1 = df.pivot_table(index=['Parcel', 'N'], 
                     columns='Imprv_Attribute', 
                     values=['Imprv_Attr_Desc', 'Imprv_Attr_Units'],
                     aggfunc='first')

df1.columns = [x[1] if x[0] == 'Imprv_Attr_Desc' else '_'.join(x[::-1]) for x in df1.columns]
df1 = df1.sort_index(axis=1).reset_index().drop(columns='N')

          Parcel  Bathrooms  Bathrooms_Imprv_Attr_Units      Bedrooms  Bedrooms_Imprv_Attr_Units      Exterior Wall  Exterior Wall_Imprv_Attr_Units       Floor Cov  Floor Cov_Imprv_Attr_Units
0  00002-000-000  2.0-Baths                         1.0  2-2 BEDROOMS                        1.0   13-PRE-FAB PANEL                           100.0  08-SHEET VINYL                        20.0
1  00002-000-000        NaN                         NaN           NaN                        NaN                NaN                             NaN       14-CARPET                        80.0
2  00011-000-000  3.0-Baths                         1.0  3-3 BEDROOMS                        1.0  15-CONCRETE BLOCK                            60.0       14-CARPET                       100.0
3  00011-000-000        NaN                         NaN           NaN                        NaN      20-FACE BRICK                            40.0             NaN                         NaN
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.