1

I have a complex text where I am categorizing different keywords stored in a dictionary:

    text = 'data-ls-static="1">Making Bio Implants, Drug Delivery and 3D Printing in Medicine,MEDICINE</h3>'

    sector = {"med tech": ['Drug Delivery' '3D printing', 'medicine', 'medical technology', 'bio cell']}

this can successfully find my keywords and categorize them with some limitations:

    pattern = r'[a-zA-Z0-9]+'

    [cat for cat in sector if any(x in re.findall(pattern,text) for x in sector[cat])]

The limitations that I cannot solve are:

  1. For example, keywords like "Drug Delivery" that are separated by a space are not recognized and therefore categorized.

  2. I was not able to make the pattern case insensitive, as words like MEDICINE are not recognized. I tried to add (?i) to the pattern but it doesn't work.

  3. The categorized keywords go into a pandas df, but they are printed into []. I tried to loop again the script to take them out but they are still there.

Data to pandas df:

    ind_list = []
    for site in url_list:
        ind = [cat for cat in indication if any(x in re.findall(pattern,soup_string) for x in indication[cat])]
        ind_list.append(ind)

    websites['Indication'] = ind_list

Current output:

Website                                  Sector                              Sub-sector                                 Therapeutical Area Focus URL status
0     url3.com                              [med tech]                                      []                                                 []          []         []
1     www.url1.com                    [med tech, services]                                      []                       [oncology, gastroenterology]          []         []
2     www.url2.com                    [med tech, services]                                      []                                        [orthopedy]          []         []

In the output I get [] that I'd like to avoid.

Can you help me with these points?

Thanks!

2
  • Regarding to point 3, can you give sample code how you setup the keywords into pandas df ?
    – SeaBean
    Mar 13, 2021 at 13:50
  • See my analysis result (some hints) as below.
    – SeaBean
    Mar 13, 2021 at 15:26

3 Answers 3

1

Give you some hints here the problem that can readily be spot:

  1. Why can't match keywords like "Drug Delivery" that are separated by a space ? This is because the regex pattern r'[a-zA-Z0-9]+' does not match for a space. You can change it to r'[a-zA-Z0-9 ]+' (added a space after 9) if you want to match also for a space. However, if you want to support other types of white spaces (e.g. \t, \n), you need to further change this regex pattern.

  2. Why don't support case insensitive match ? Your code fragment any(x in re.findall(pattern,text) for x in sector[cat]) requires x to have the same upper/lower case for BOTH being in result of re.findall and being in sector[cat]. This constrain even cannot be bypassed by setting flags=re.I in the re.findall() call. Suggest you to convert them all to the same case before checking. That is, for example change them all to lower cases before matching: any(x in re.findall(pattern,text.lower()) for x.lower() in sector[cat]) Here we added .lower() to both text and x.lower().

With the above 2 changes, it should allow you to capture some categorized keywords.

Actually, for this particular case, you may not need to use regular expression and re.findall at all. You may just check e.g. sector[cat][i].lower()) in text.lower(). That is, change the list comprehension as follows:

[cat for cat in sector if any(x in text.lower() for x in [y.lower() for y in sector[cat]])]

Edit

Test Run with 2-word phrase:

text = 'drug delivery'
sector = {"med tech": ['Drug Delivery', '3D printing', 'medicine', 'medical technology', 'bio cell']}
[cat for cat in sector if any(x in text.lower() for x in [y.lower() for y in sector[cat]])]

Output:       # Successfully got the categorizing keyword even with dictionary values of different upper/lower cases
['med tech']

text = 'Drug Store fast delivery'
[cat for cat in sector if any(x in text.lower() for x in [y.lower() for y in sector[cat]])]

Ouptput:    # Correctly doesn't match with extra words in between 

[]
18
  • #SeaBean thanks for the hints. Yes, I solved the insensitive issue. The space thing still is not working, meaning that a dictionary value like "Drug Delivery" matches for "Drug" or "Delivery" and is not stringent enough,
    – Steven
    Mar 13, 2021 at 16:09
  • @Steven This is exactly the idea in my paragraph "Actually..." that you can test for eg. 'Drug Delivery' in text so that you can check for the whole phrase (2 words) in the target string (called text or soup_string in your code.
    – SeaBean
    Mar 13, 2021 at 17:15
  • @Steven Slightly amended the sample code of list comprehension in the bottom of my answer. This should better demonstrate what I said.
    – SeaBean
    Mar 13, 2021 at 17:28
  • 1
    @Steven, it's still not clear enough. There is no column named Indication in your sample data. As you have tested with small data that my latest code works to extract values into the list ind, the problem should be lying in other parts of your codes. Suggest to do a divide-and-conquer test to start with one website that should have some matches and then check for values extracted into ind then trace your program to see why its values can't go into the dataframe.
    – SeaBean
    Mar 14, 2021 at 21:57
  • 1
    @Steven Great to hear you are almost done! For what you mentioned you get [med tech] instead of just 'med tech', I think it is because your extracted result in each loop iteration is in ind which is a list (hence [...] for it is a list). If you want to just get string, you can concatenate the strings stored in the list ind before appending to ind_list. For example, use codes like: ind_text = ', '.join(ind) and then ind_list.append(ind_text) to replace ind_list.append(ind)
    – SeaBean
    Mar 15, 2021 at 7:41
1

Can you try a different approach other than regex,
I would suggest difflib when you have two similar matching words.

1

findall is pretty wasteful here since you are repeatedly breaking up the string for each keyword.

If you want to test whether the keyword is in the string:

[cat for cat in sector if any(re.search(word, text, re.I) for word in sector[cat])]
# Output: med tech
3
  • thanks for the hint. It seems working better, but it matches single words like "Drug" or "Delivery" in the text when in the dictionary I want to find only "Drug Delivery".
    – Steven
    Mar 13, 2021 at 13:57
  • Can you add your test case? This text doesn't match anything: text = 'data-ls-static="1">Drug</h3>' Mar 13, 2021 at 14:16
  • I updated the text and dictionary. In the text you can find "Bio Implants", in the dictionary one of the value is " bio cell". In this situation it matches the key med tech because it found "bio", but in reality I wanted to match the exact value "bio cell" if any. In this case it should not match anything.
    – Steven
    Mar 13, 2021 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.