0

I want to search all hyperlink that its text name includes "article" in https://www.geeksforgeeks.org/ for example, on the bottom of this webpage

Write an Article
Improve an Article

I want to get them all hyperlink and print them, so I tried to,

from urllib.request import urlopen
from bs4 import BeautifulSoup
import os
import re

url = 'https://www.geeksforgeeks.org/'

reqs = requests.get(url)
soup = BeautifulSoup(reqs.text, "html.parser")
links = []
for link in soup.findAll('a',href = True):
    #print(link.get("href")

    if re.search('/article$', href):
        links.append(link.get("href"))

However, it get a [] in result, how to solve it?

4
  • Hi, do you have part of your html so we can reproduce your code? That would make it easier to debug and help you with your question. Mar 18, 2022 at 7:15
  • 1
    @3dSpatialUser check it again, please
    – 4daJKong
    Mar 18, 2022 at 7:25
  • I’m voting to close this question because provided code do not lead to your issue with empty ResultSet, there are several other issues (module, variables) - Just a hint: If you wanna find article in text, why are you searching in href?
    – HedgeHog
    Mar 18, 2022 at 7:41
  • @HedgeHog the hyperlink's name is "article" in website
    – 4daJKong
    Mar 18, 2022 at 7:45

1 Answer 1

1

Here is something you can try: Note that there are more links with the test article in the link you provided, but it gives the idea how you can deal with this.

In this case I just checked if the word article is in the text of that tag. You can use regex search there, but for this example it is an overkill.

import requests
from bs4 import BeautifulSoup

url = 'https://www.geeksforgeeks.org/'
res = requests.get(url)

if res.status_code != 200:
    'no resquest'

soup = BeautifulSoup(res.content, "html.parser")

links_with_article = soup.findAll(lambda tag:tag.name=="a" and "article" in tag.text.lower())

EDIT:

If you know that there is a word in the href, i.e. in the link itself:

soup.select("a[href*=article]")

this will search for the word article in the href of all elements a.

Edit: get only href:

hrefs = [link.get('href') for link in links_with_article]
5
  • Thanks a lot, that's the thing I want, but it return all info, how can I only get hype?like,results = soup.findAll(zone_links.get("href"))
    – 4daJKong
    Mar 18, 2022 at 7:58
  • please see my edit Mar 18, 2022 at 8:07
  • Thank you so much, I still have a little question, how can I use regex to rewrite it? I mean, for example "4article", "5article"... I write re.match ("/darticle$", tag.text.lower()) why not true?
    – 4daJKong
    Mar 18, 2022 at 8:38
  • Your /d should be \d. For example you can use: links_with_article = soup.findAll(lambda tag:tag.name=="a" and re.search(r'\darticle', tag.text.lower())) to find the article. I used search to search the whole string, match searched only from the start of the string. Mar 18, 2022 at 8:45
  • Thanks for your help again and I really appreciate it!
    – 4daJKong
    Mar 18, 2022 at 15:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.