I am trying to model the score that a post receives, based on both the text of the post, and other features (time of day, length of post, etc.)
I am wondering how to best combine these different types of features into one model. Right now, I have something like the following (stolen from here and here).
import pandas as pd
...
def features(p):
terms = vectorizer(p[0])
d = {'feature_1': p[1], 'feature_2': p[2]}
for t in terms:
d[t] = d.get(t, 0) + 1
return d
posts = pd.read_csv('path/to/csv')
# Create vectorizer for function to use
vectorizer = CountVectorizer(binary=True, ngram_range=(1, 2)).build_tokenizer()
y = posts["score"].values.astype(np.float32)
vect = DictVectorizer()
# This is the part I want to fix
temp = zip(list(posts.message), list(posts.feature_1), list(posts.feature_2))
tokenized = map(lambda x: features(x), temp)
X = vect.fit_transform(tokenized)
It seems very silly to extract all of the features I want out of the pandas dataframe, just to zip them all back together. Is there a better way of doing this step?
The CSV looks something like the following:
ID,message,feature_1,feature_2
1,'This is the text',4,7
2,'This is more text',3,2
...