1

how to crawl the first paragraph link in wiki?

All the links in the parentheses should be excluded. As an example i provide the follow link:

https://en.wikipedia.org/wiki/Data.

On this page the first link i want to crawl is "qualitative" (href="/wiki/Qualitative_property"). My code has excluded all the special links like footnotes and pronunciation but can't exclude the normal link in the parentheses.

import requests
from bs4 import BeautifulSoup
response = requests.get('https://en.wikipedia.org/wiki/Data')
html = response.text
soup = BeautifulSoup(html, "html.parser")
link = soup.find(id='mw-content-text').find(class_="mw-parser-output").find_all('p', recursive=False)
list_a = []
for element in link:
    if element.find("a", recursive=False):
        print(element.find("a", recursive=False).get('href'))
        break
1
  • I think I fixed your problem. Take a look at my answer!
    – JoshuaCS
    Jan 14, 2019 at 14:39

1 Answer 1

1

Well, technically speaking, those links are not different from the links outside the parentheses. If you look closer at the href attribute of those links, all of them begin with /wiki/Help: so, you can leave them out if that happens. In the code below I used regular expressions for doing that:

Code

import re
import requests
from bs4 import BeautifulSoup
response = requests.get('https://en.wikipedia.org/wiki/Data')
html = response.text
soup = BeautifulSoup(html, "html.parser")
paragraphs = soup.find(id='mw-content-text').find(class_="mw-parser-output").find_all('p', recursive=False)
list_a = []

# Help links RegEx
help_link_regex = re.compile('^/wiki/Help:')

for p in paragraphs:
    p_links = p.find_all("a", recursive=False)

    for link in p_links:
        # Leave them out if they match the previous RegEx
        if not help_link_regex.match(link.get('href')):
            print(link.get('href'))
            list_a.append(link.get('href'))
            break

Output

/wiki/Qualitative_property
/wiki/Information
/wiki/Measurement
/wiki/Data_(word)
/wiki/Information
/wiki/Knowledge
/wiki/Sign
/wiki/Marketing
/wiki/Analog_computer
/wiki/Johanna_Drucker

Note the first link in this list is the first link (outside parentheses) in the first paragraph: the link you wanted.

The previous code just adds the first non-help link of each paragraph to list_a, if you want to get them all, just remove the break:

Output (after removing the break)

/wiki/Qualitative_property
/wiki/Quantitative_data
/wiki/Variable_(research)
/wiki/Information
/wiki/Scientific_research
/wiki/Stock_price
/wiki/Crime_rate
/wiki/Unemployment_rate
/wiki/Literacy
/wiki/Homelessness
/wiki/Measurement
/wiki/Data_reporting
/wiki/Data_analysis
/wiki/Data_visualization
/wiki/Concept
/wiki/Information
/wiki/Knowledge
/wiki/Data_processing
/wiki/Number
/wiki/Character_(computing)
/wiki/Outlier
/wiki/Field_work
/wiki/In_situ
/wiki/Experimental_data
/wiki/Petroleum
/wiki/Digital_economy
/wiki/Data_(word)
/wiki/Mass_noun
/wiki/Information
/wiki/Knowledge
/wiki/Wisdom
/wiki/Shannon_entropy
/wiki/Knowledge
/wiki/Mount_Everest
/wiki/Altimeter
/wiki/Sign
/wiki/Marketing
/wiki/Social_services
/wiki/Truth
/wiki/Analog_computer
/wiki/Computer
/wiki/Alphabet
/wiki/Computer_program
/wiki/Lisp_(programming_language)
/wiki/Metadata
/wiki/Johanna_Drucker

I hope this helps you, otherwise, let me know what went wrong.

1
  • @joe please, let me know if this works for you. If so, mark my answer as accepted (green checkmark) and upvote it. Thanks in advance
    – JoshuaCS
    Jan 14, 2019 at 14:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.