I am dealing with some aggregated membership data from multiple sources. I have names in one column, and a long membership memos string in the other. I would like the best way to match the name into membership memo.
For example,
I would like the best way to find:
'Barack Obama'
in the following strings, since this data is aggregated and the formats may be different. here are a few examples:
"Member Data REWNEW:EX PAID ID:234242 Barack Obama WASHINGTON DC LAST CO 2834298:EEXE:00WIE"
"Member Data REWNEW:EX PAID ID:234242 Barack Hussein Obama WASHINGTON DC LAST CO 2834298:EEXE:00WIE"
"Member Data REWNEW:EX PAID ID:234242 Obama Barack WASHINGTON DC LAST CO 2834298:EEXE:00WIE"
"Member Data REWNEW:EX PAID ID:234242 Barack H Obama WASHINGTON DC LAST CO 2834298:EEXE:00WIE"
In the past, I've used fuzzywuzzy to do fuzzy logic matching. This tends to work well if I am comparing two strings, but not trying to find one string within another longer string. For example:
from fuzzywuzzy import fuzz
from fuzzywuzzy import fuzz
print(fuzz.ratio("Barack Obama", "Barack Obama"))
print(fuzz.ratio("Michelle Obama", "Barack Obama"))
print(fuzz.ratio("Barack Obama", "Member Data REWNEW:EX PAID ID:234242 Barack Hussein Obama WASHINGTON DC LAST CO 2834298:EEXE:00WIE"))
print(fuzz.ratio("Michelle Obama", "Member Data REWNEW:EX PAID ID:234242 Barack Hussein Obama WASHINGTON DC LAST CO 2834298:EEXE:00WIE"))
100
54
22
16
It's very clear that the first two are very different (100 vs 54) however the second two do not appear to be all that different and thus this is not the best method.
Does anyone have any ideas on how best to implement this type of string search?
Thank you so much!
UPDATE1 :
I tried:
memo_string="Member Data REWNEW:EX PAID ID:234242 Barack Hussein Obama WASHINGTON DC LAST CO 2834298:EEXE:00WIE"
search_terms = "Barack Obama"
memo_words = memo_string.split(" ")
search_term_count = len(search_terms.split(" "))
memo_slices = []
for i in range(len(memo_words) - search_term_count):
memo_slices.append(" ".join(memo_words[i:i+search_term_count]))
max_for_memo = 0
best_match_in_memo = None
for memo_slice in memo_slices:
fuzz_score = fuzz.ratio(search_terms, memo_slice)
if fuzz_score > max_for_memo:
max_for_memo = fuzz_score
best_match_in_memo = memo_slice
print(max_for_memo)
I also tried with search_terms = 'Michelle Obama'
The two scores were 52 and 50, respectively, which still presents the challenge that I stated in the beginning, which is that I would like some sort of logic that separates the two more definitively.
Thoughts?
Thank you so much!