As in sklearn, LogisticRegression(short for LR) has not direct method for solving weighted LR, so i pass to SGDClassifier(SGD).
As with my experiment: i generate data follow LR distribution with parametre intercept=0, beta=2. And run LR and SGD to estimate them. To compare this two, i set the same penalty parameter(my original idea is setting them to 0, but as they can't be setted to 0, i give big C for LR and small alpha for SGD )
As i see, LR almost do well for estimation, but it's difficult to setting the parametre for SGD. The main problem is the choice of eta0 and the learning_rate: 'constant' (too slow), 'optimal' or 'invscaling'.
My idea is watching the loss function, if it's likely to go down, increase the n_iter. if it go down too slow, increase the eta0. But
1.how to return the value of loss function for each epoch, i see them by change verbose to 1, but i don't know how to get return the value. (may be partiel_fit?)
2.is there more intelligent (automatique) way to this work? if not i should relance the training process many times And more complicated if i use cross validation
Thanks you for all your advice. If i was not clear, please let me know.
P.S. the code in Python require indent block, as i'm new to stackoverflow, i don't know how to do this, so if you want to execute de code, please add indented block after the def.
import random
import numpy as np
from sklearn.linear_model import LogisticRegression,SGDClassifier
def simule_logistic(n):
beta=0.2
x=[]
seuil=[]
for i in range(n):
x.append(random.normalvariate(1, 2))
seuil.append(random.uniform(0, 1))
x=np.array(x)
seuil=np.array(seuil)
p=1.0/(1+ np.exp(-x*beta))
y=[]
for i in range(n):
if p[i]<seuil[i]:
y.append(0)
else:
y.append(1)
y=np.array(y)
return x, p,y
if __name__=='__main__':
n=100000
x,p,y=simule_logistic(n)
x=x.reshape((n,1))
print x.shape
print y.shape
l=LogisticRegression(C=1000000,penalty='l1')
l.fit(x,y)
sgd=SGDClassifier(n_iter=100,n_jobs=1, loss='log',alpha=1.0/1000000,l1_ratio=1,learning_rate='optimal',eta0=0.01)
print sgd
sgd.fit(x,y)
#methode regression
print 'l',l.coef_
print l.intercept_
print 'sgd',sgd.coef_
print sgd.intercept_