-1

I have to fix a problem related to a large number of user generated skills.
Users can put any skills on their profiles and i want to merge the ones that are the same:
I have this pairs (among others):
React, React Js, React.js, reactjs
MS Office, Microsoft Office MS Word, Microsoft Word

I tried the Levenshtein algorithm and different spell checkers but they do not work in such cases.

Does anyone know a solution for a problem like this?

1
  • 1
    How exactly didn't the Levenshtein distance work? Was the edit distance larger than expected?
    – Codor
    Oct 29, 2018 at 9:19

1 Answer 1

0

The way I see this problem, there are 2 solutions:

  • build an exhaustive list of all possible skills you consider equal
  • define a suitable 'distance' between two given skills and define a threshold you feel yields good (enough) results

Suppose you are going with the first option, then the pseudocode is pretty simple:

  1. for any list of skills the user enters, loop over each skill S
  2. if S is in the set of 'accepted' skills, then keep S
  3. if S does not belong to the set of 'accepted' skills, check whether an accepted skill S2 has S as one of its variants. If it does, return S2

Suppose you are going with the second option:

  1. for any list of skills the user enters, loop over each skill S
  2. set S to uppercase
  3. split S by any token that is not [A-Z], name this Sp
  4. sort Sp alphabetically
  5. re-join the tokens in Sp by space, name this Sc
  6. use the Levenshtein distance to compare Sc to other items in the list of skills

e.g.

React, React Js, React.js, reactjs

React --> REACT --> [REACT] --> [REACT] --> REACT
React Js --> REACT JS --> [REACT JS] --> [JS REACT] --> JS REACT // distance 3
React.js --> REACT.JS --> [REACT JS] --> [JS REACT] --> JS REACT // distance 0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.