I am trying to convert categorical data into binary to be able to classify with an algorithm like logistic regression. I thought of using OneHotEncoder from 'sklearn.preprocessing' module but the problem is the dataframe entries are A, B pairs of arrays with different lengths, each row has pair of same-length arrays not equal to array lengths in other rows. OneHotEncoder does not accept dataframe like mine
In [34]: data.index
Out[34]: Index([train1, train2, train3, ..., train7829, train7830, train7831], dtype=object)
In [35]: data.columns
Out[35]: Index([A, B], dtype=object)
SampleID A B
train1 [2092.0, 1143.0, 390.0, ...] [5651.0, 4449.0, 4012.0...]
train2 [3158.0, 3158.0, 3684.0, 3684.0....] [2.0, 4.0, 2.0, 1.0...]
train3 [1699.0, 1808.0 ,...] [0.0, 1.0...]
So, I want to highlight again that each A and B pair has the same length but the length is variable across different pairs. Dataframe contains numerical, categorical and binary values. I have another csv file with the information about every entry type. I read the file filter out categorical entries in both columns like this:
info=data_io.read_train_info()
col1=info.columns[0]
col2=info.columns[1]
info=info[(info[col1]=='Categorical')&(info[col2]=='Categorical')]
Then I use info.index
to filter my training dataframe
filtered = data.loc[info.index]
Than I wrote an utility function to change dimensions of each array so that I can encode them later
def setDim(df):
for item in x[x.columns[0]].index:
df[df.columns[0]][item].shape=(1,df[df.columns[0]][item].shape[0])
df[df.columns[1]][item].shape=(1,df[df.columns[1]][item].shape[0])
setDim(filtered)
Then I thought to combine each pair of arrays into 2-row matrix so that I can pass it to encoder then to separate them again after encoding, like this:
import numpy as np
from sklearn.preprocessing import OneHotEncoder
def makeSparse(df):
enc = OneHotEncoder()
for i in df.index:
cd=np.append(df['A'][i],df['B'][i],axis=0)
a=enc.fit_transform(cd)
df['A'][i] = a[0,:]
df['B'][i] = a[1,:]
makeSparse(filtered)
After all these steps get a sparse dataframe. My questions are:
- is this a right way to encode this dataframe?(I highly doubt it)
- if no, then what alternatives do you offer?
Thanks a lot for your time helping me.