-1

Hi everyone i am parsing an html doc with beautifulsoup. However, one area of information I cant seem to parse:

the html:

<small>
<span class="label label-primary">CVE-2019-11198</span>
<span class="label label-warning">6.1 - Medium</span>
- August 05, 2019
</small>

I am parsing this whole block, but want to parse the CVE-2019-11198 , 6.1 , Medium , and August 05, 2019 as separate values. Instead im getting the whole block under <small> with the following code:

original:

cves=soup.find_all("div", class_="cve_listing")
for cve in cves:
    #CVE, vuln numeric rating, vuln sev cat, vuln date
    vulninfo=cve.find("small").text

updated:

cves=soup.find_all("div", class_="cve_listing")
for cve in cves:
    vulncve=cve.find("span", class_="label-primary")
    vulninfo=cve.select_one('span.label').parent
    vulninfores=[x.get_text(strip=True) for x in vulninfo.contents if len(x.text) > 1]

outputs:

AttributeError: 'NavigableString' object has no attribute 'text'

any thoughts on how to parse this efficiently?

3
  • Use the <span> classes to get their text, then remove them from vulinfo and parse what's left?
    – MattDMo
    Sep 6, 2022 at 15:07
  • @MattDMo - the problem is doing that its not parsing anything. For example: vulncve=cve.find("span", class_="label label-primary").text
    – Jshee
    Sep 6, 2022 at 15:11
  • Can you give us the URL you're trying to parse? Alternatively, is there an API you can use or an RSS feed you can parse, something like that?
    – MattDMo
    Sep 6, 2022 at 15:15

2 Answers 2

1

You need a bit modify your question.

  1. You have selected "div", class_="cve_listing" but didn't show the html

  2. You can't invoke get_text() and contents method at the same time. Try the below code:

Example:

cves=soup.find_all("div", class_="cve_listing")
for cve in cves:
    vulncve=cve.find("span", class_="label-primary")
    vulninfo=cve.select_one('span.label')
    vulninfores=[x.get_text(strip=True) for x in soup.select(".cve_listing small")][-1]
   
1

Not having the url of the actual page means I cannot test it, but supposing the html is correct and you can reach it as stated in your question, this is one way of getting that info:

from bs4 import BeautifulSoup

html = '''
<small>
<span class="label label-primary">CVE-2019-11198</span>
<span class="label label-warning">6.1 - Medium</span>
- August 05, 2019
</small>
'''

soup = BeautifulSoup(html, 'html.parser')
data = soup.select_one('span.label').parent
desired_result = [x.get_text(strip=True) for x in data.contents if len(x.text) > 1]
print(desired_result)

Result:

['CVE-2019-11198', '6.1 - Medium', '- August 05, 2019']

BeautifulSoup documentation: https://beautiful-soup-4.readthedocs.io/en/latest/index.html

5
  • Thanks for this, but im looking to parse the result into separate results - e.g. result1=cve-2019-111986.1, result2=medium, etc
    – Jshee
    Sep 6, 2022 at 15:23
  • Give me a moment @Jshee... Sep 6, 2022 at 15:24
  • @Jshee now you have a list with 3 (separate) items. Is this what you are after? Also, let me know if you need a dictionary (and if you are comfortable with transforming a list into a dict) -- or a dataframe Sep 6, 2022 at 15:28
  • Hi @platipus_on_fire - please see updated code and the error I'm getting
    – Jshee
    Sep 6, 2022 at 15:40
  • @Jshee the code works. Try and update your packages, do pip install -U bs4, it should sort out your issue. Sep 6, 2022 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.