2

Currently I have a database consisted of about 600,000 records represents merchandise with their category information look like below:

{'title': 'Canon camera', 'category': 'Camera'},
{'title': 'Panasonic regrigerator', 'category': 'Refrigerator'},
{'title': 'Logo', 'category': 'Toys'},
....

But there are merchandises without category information.

{'title': 'Iphone6', 'category': ''},

So I'm thinking whether it is possible to train a text classifier based on my items' name by using scikit-learn to help me predict which the category should the merchandise be. I'm forming this problem as a multi-class text classification but there are also one~many pictures for each item so maybe deep learning/Keras can also be used?

I don't know what is the best way to solve this problem so any suggestion or advice is welcome, thank you for reading this.

P.S. the actual text is in Japanese

2
  • how many times does the title 'iphone6 appear, and does it have other categories associated to it? if it is is the only entry you will have to resort to an external training source
    – Y K
    Mar 7, 2017 at 12:49
  • @yosemite_k Thanks for the reply. I think there is small chance that there will appear two identical titles in item but there will be reappearing terms in multiple items' title. I will provide more information later.
    – Meng Lee
    Mar 7, 2017 at 13:06

1 Answer 1

1

You could build a 2-char / 3-char model and calculate values e.g. how often does the 3-gram "pho" appear in the category "Camera".

trigrams = {}
for record in records:    # only the ones with categories
    title = record['title']
    cat = record['category']
    for trigram in zip(title, title[1:], title[2:])
        if trigram not in trigrams:
            trigrams[trigram] = {}
            for category in categories:
                trigrams[trigram] = 0
        trigrams[trigram][cat] += 1

Now you can use the titles trigrams to calculate a score:

scores = []
for trigram in zip(title, title[1:], title[2:]):
    score = []
    for cat in categories:
       score.append(trigrams[trigram][cat])
    # Normalize
    sum_ = float(sum(score))
    score = [s / sum_ for s in score]
    scores.append(score)

Now score contains a probability distribution for every trigram: P(class | trigram). It does not take into account that some classes are just more common (prior, see Bayes theorem). I'm currently also not quite sure if you should do something against the problem that some titles might just be really long and thus have a lot of trigrams. I guess taking the prior does that already.

If it turns out that you have many trigrams missing, you could switch to bigrams. Or simply do Laplace smoothing.

edit: I've just seen that the text is in Japanese. I think the n-gram approach might be useless there. You could translate the name. However, it is probably easier to just take other sources for this information (e.g. wikipedia / amazon / ebay?)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.