24

I am trying to model the score that a post receives, based on both the text of the post, and other features (time of day, length of post, etc.)

I am wondering how to best combine these different types of features into one model. Right now, I have something like the following (stolen from here and here).

import pandas as pd
...

def features(p):
    terms = vectorizer(p[0])
    d = {'feature_1': p[1], 'feature_2': p[2]}
    for t in terms:
        d[t] = d.get(t, 0) + 1
    return d

posts = pd.read_csv('path/to/csv')

# Create vectorizer for function to use
vectorizer = CountVectorizer(binary=True, ngram_range=(1, 2)).build_tokenizer()
y = posts["score"].values.astype(np.float32) 
vect = DictVectorizer()

# This is the part I want to fix
temp = zip(list(posts.message), list(posts.feature_1), list(posts.feature_2))
tokenized = map(lambda x: features(x), temp)
X = vect.fit_transform(tokenized)

It seems very silly to extract all of the features I want out of the pandas dataframe, just to zip them all back together. Is there a better way of doing this step?

The CSV looks something like the following:

ID,message,feature_1,feature_2
1,'This is the text',4,7
2,'This is more text',3,2
...
3
  • Can you show a sample of your csv?
    – elyase
    Jun 6, 2015 at 22:05
  • @elyase, I just added a toy version of it.
    – Jeremy
    Jun 7, 2015 at 3:09
  • Your code is not stolen, it is inspired from here and there Mar 5 at 13:40

1 Answer 1

29

You could do everything with your map and lambda:

tokenized=map(lambda msg, ft1, ft2: features([msg,ft1,ft2]), posts.message,posts.feature_1, posts.feature_2)

This saves doing your interim temp step and iterates through the 3 columns.

Another solution would be convert the messages into their CountVectorizer sparse matrix and join this matrix with the feature values from the posts dataframe (this skips having to construct a dict and produces a sparse matrix similar to what you would get with DictVectorizer):

import scipy as sp
posts = pd.read_csv('post.csv')

# Create vectorizer for function to use
vectorizer = CountVectorizer(binary=True, ngram_range=(1, 2))
y = posts["score"].values.astype(np.float32) 

X = sp.sparse.hstack((vectorizer.fit_transform(posts.message),posts[['feature_1','feature_2']].values),format='csr')
X_columns=vectorizer.get_feature_names()+posts[['feature_1','feature_2']].columns.tolist()


posts
Out[38]: 
   ID              message  feature_1  feature_2  score
0   1   'This is the text'          4          7     10
1   2  'This is more text'          3          2      9
2   3   'More random text'          3          2      9

X_columns
Out[39]: 
[u'is',
 u'is more',
 u'is the',
 u'more',
 u'more random',
 u'more text',
 u'random',
 u'random text',
 u'text',
 u'the',
 u'the text',
 u'this',
 u'this is',
 'feature_1',
 'feature_2']

X.toarray()
Out[40]: 
array([[1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 4, 7],
       [1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 1, 3, 2],
       [0, 0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 3, 2]])

Additionally sklearn-pandas has DataFrameMapper which does what you're looking for too:

from sklearn_pandas import DataFrameMapper
mapper = DataFrameMapper([
    (['feature_1', 'feature_2'], None),
    ('message',CountVectorizer(binary=True, ngram_range=(1, 2)))
])
X=mapper.fit_transform(posts)

X
Out[71]: 
array([[4, 7, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
       [3, 2, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 1],
       [3, 2, 0, 0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0]])

Note:X is not sparse when using this last method.

X_columns=mapper.features[0][0]+mapper.features[1][1].get_feature_names()

X_columns
Out[76]: 
['feature_1',
 'feature_2',
 u'is',
 u'is more',
 u'is the',
 u'more',
 u'more random',
 u'more text',
 u'random',
 u'random text',
 u'text',
 u'the',
 u'the text',
 u'this',
 u'this is']
3
  • 1
    thanks @khammel i put this into a gist gist.github.com/danemacaulay/c8e3194b63570de1cf88f431ade32107 Oct 5, 2017 at 20:39
  • 1
    Thank you a lot. If I saw this earlier, it would spare me of losing 4 hours trying to merge dense matrix of tf-idf features with existing features read from csv (repetitively getting MemoryError). Sep 9, 2018 at 17:31
  • @khammel Do I need to apply this function for test set as well? or just pass the X_test to predict function? Sep 25, 2021 at 17:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.