1

I have been trying to self host with apache an sklearn classifier that I put together, and I ended up using joblib to serialize the saved model, then load it in a flask app. Now, this app worked perfectly when running flask's built in development server, but when I set this up with a debian 9 apache server, I get a 500 error. Delving into apache's error.log, I get:

AttributeError: module '__main__' has no attribute 'tokenize'

Now, this is funny to me because while I did write my own tokenizer, the web app gave me no problems when I was running it locally. Furthermore, the saved model that I used was trained on the webserver, so slightly different library versions should not be a problem.

My code for the web app is:

import re
import sys

from flask import Flask, request, render_template
from nltk import word_tokenize
from nltk.stem.wordnet import WordNetLemmatizer
from sklearn.externals import joblib

app = Flask(__name__)



def tokenize(text):
    # text = text.translate(str.maketrans('','',string.punctuation))
    text = re.sub(r'\W+', ' ', text)
    tokens = word_tokenize(text)
    lemas = []
    for item in tokens:
        lemas.append(WordNetLemmatizer().lemmatize(item))
    return lemas

@app.route('/')
def home():
    return render_template('home.html')

@app.route('/analyze',methods=['POST','GET'])
def analyze():
    if request.method=='POST':
        result=request.form
        input_text = result['input_text']

        clf = joblib.load("model.pkl.z")
        parameters = clf.named_steps['clf'].get_params()
        predicted = clf.predict([input_text])
        # print(predicted)
        certainty = clf.decision_function([input_text])

        # Is it bonkers?
        if predicted[0]:
            verdict = "Not too nuts!"
        else:
            verdict = "Bonkers!"

        return render_template('result.html',prediction=[input_text, verdict, float(certainty), parameters])

if __name__ == '__main__':
    #app.debug = True
    app.run()

With the .wsgi file being:

import sys 
sys.path.append('/var/www/mysite')

from conspiracydetector import app as application

Furthermore, I trained the model with this code:

import logging
import pprint  # Pretty stuff
import re
import sys  # For command line arguments
from time import time  # to show progress

import numpy as np
from nltk import word_tokenize
from nltk.stem.wordnet import WordNetLemmatizer
from sklearn import metrics
from sklearn.datasets import load_files
from sklearn.externals import joblib  # In order to save
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.svm import LinearSVC

# Tokenizer that does stemming and strips punctuation
def tokenize(text):
    # text = text.translate(str.maketrans('','',string.punctuation))
    text = re.sub(r'\W+', ' ', text)
    tokens = word_tokenize(text)
    lemas = []
    for item in tokens:
        lemas.append(WordNetLemmatizer().lemmatize(item))
    return lemas

if __name__ == "__main__":
    # NOTE: we put the following in a 'if __name__ == "__main__"' protected
    # block to be able to use a multi-core grid search that also works under
    # Windows, see: http://docs.python.org/library/multiprocessing.html#windows
    # The multiprocessing module is used as the backend of joblib.Parallel
    # that is used when n_jobs != 1 in GridSearchCV

    # Display progress logs on stdout
    print("Initializing...")
    # Command line arguments
    save = sys.argv[1]
    training_directory = sys.argv[2]

    logging.basicConfig(level=logging.INFO,
                        format='%(asctime)s %(levelname)s %(message)s')

    dataset = load_files(training_directory, shuffle=False)
    print("n_samples: %d" % len(dataset.data))

    # split the dataset in training and test set:
    print("Splitting the dataset in training and test set...")
    docs_train, docs_test, y_train, y_test = train_test_split(
        dataset.data, dataset.target, test_size=0.25, random_state=None)

    # Build a vectorizer / classifier pipeline that filters out tokens
    # that are too rare or too frequent
    # Also remove stop words
    print("Loading list of stop words...")
    with open('stopwords.txt', 'r') as f:
        words = [line.strip() for line in f]

    print("Stop words list loaded...")
    print("Setting up pipeline...")
    pipeline = Pipeline(
        [
            # ('vect', TfidfVectorizer(stop_words=words, min_df=0.001, max_df=0.5, ngram_range=(1,1))),
            ('vect',
             TfidfVectorizer(tokenizer=tokenize, stop_words=words, min_df=0.001, max_df=0.5, ngram_range=(1, 1))),
            ('clf', LinearSVC(C=5000)),
        ])

    print("Pipeline:", [name for name, _ in pipeline.steps])

    # Build a grid search to find out whether unigrams or bigrams are
    # more useful.
    # Fit the pipeline on the training set using grid search for the parameters
    print("Initializing grid search...")

    # uncommenting more parameters will give better exploring power but will
    # increase processing time in a combinatorial way
    parameters = {
        # 'vect__ngram_range': [(1, 1), (1, 2)],
        # 'vect__min_df': (0.0005, 0.001),
        # 'vect__max_df': (0.25, 0.5),
        # 'clf__C': (10, 15, 20),
    }
    print("Parameters:")
    pprint.pprint(parameters)
    grid_search = GridSearchCV(
        pipeline,
        parameters,
        n_jobs=-1,
        verbose=True)

    print("Training and performing grid search...\n")
    t0 = time()
    grid_search.fit(docs_train, y_train)
    print("\nDone in %0.3fs!\n" % (time() - t0))

    # Print the mean and std for each candidate along with the parameter
    # settings for all the candidates explored by grid search.
    n_candidates = len(grid_search.cv_results_['params'])
    for i in range(n_candidates):
        print(i, 'params - %s; mean - %0.2f; std - %0.2f'
              % (grid_search.cv_results_['params'][i],
                 grid_search.cv_results_['mean_test_score'][i],
                 grid_search.cv_results_['std_test_score'][i]))

    # Predict the outcome on the testing set and store it in a variable
    # named y_predicted
    print("\nRunning against testing set...\n")
    y_predicted = grid_search.predict(docs_test)

    # Save model
    print("\nSaving model to", save, "...")
    joblib.dump(grid_search.best_estimator_, save)
    print("Model Saved! \nPrepare for some awesome stats!")

I must confess that I am pretty stumped, and after tinkering around, searching, and making sure that my server is configured correctly, I felt that perhaps someone here might be able to help. Any help is appreciated, and if there is any more information that I need to provide, please let me know and I will be happy to.

Also, I am running:

  • python 3.5.3 with nltk and sklearn.
1
  • Is there any more information that I need to add?
    – Fake Name
    Aug 12, 2017 at 0:04

1 Answer 1

1

I solved this problem, although imperfectly, by removing my custom tokenizer and falling back on one of sklearn's.

However, I am still in the dark on how to integrate my own tokenizer.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.