1

I would like to know what kind of approach i need to adopt with Python or Flask for do the following task :

  • check to see if the url is valid
  • if valid return a list of all links on that page and its sub-pages

My editor is sublime and i run it under Windows Powershell

Now my code show this :

enter image description here

So when you input a search it go o a new page and show the result ( for exemple:ddddd)

enter image description here

BUT i want to check if the URL is valid or not and if valid return a list of all links on that page and its sub-pages like this :

enter image description here

Any idea for a Newbie in the World of Programming ?(not very new now,been still have a lot to learn..)

Thanks for the Help.

Here my code who bring this result (it's Work) :

So a project folder with inside my .py set with Flask and a templates folder with the .html.


Python file

# -*- coding: utf-8 -*-
from flask import Flask, render_template, request
import re

app = Flask (__name__)

@app.route("/")
def index():
    return render_template('index.html')

@app.route('/search', methods=['POST', 'GET'])
def search():
    error = True
    if request.method == 'POST':
        return request.form['urlsearch']
    else:
        return request.args.get('urlsearch')

if __name__ == "__main__":
    app.run()

HTML FILE

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html lang="en">
<head>
    <title>URL TEST</title>
</head>
<body>
    <ul id="navigation">
    {% for item in navigation %}
        <li><a href="{{ item.href }}">{{ item.caption }}</a></li>
    {% endfor %}
</ul>

    <h1 style="color:orange;">You can put your URL here :</h1>
    {{ a_variable }}

<form method="get" action="/search">
    <p>Please Input an URL below : </p>
<input type="text" name="urlsearch" />
        <input type="submit" value="Search" />
    </form>
</body>
</html>

2 Answers 2

5

You can use mechanize:

from mechanize import Browser

br = Browser()    
r = br.open("http://www.example.com/")

if r.code == 200:
    for link in br.links():
        print link
else:
    print "Error loading page"

Or urllib2 and BeautifulSoup

from BeautifulSoup import BeautifulSoup
import urllib2

html_page = urllib2.urlopen("http://www.example.com")
if html_page.getcode() == 200:
    soup = BeautifulSoup(html_page)
    for link in soup.findAll('a'):
        print link.get('href')
else:
    print "Error loading page"

I haven't worked much with Flask before, but try this:

As I understand urlsearch is the URL that you are getting from form, so add check of it

@app.route('/search', methods=['POST', 'GET'])
def search():
    error = True
    if request.method == 'POST':
        return request.form['urlsearch']
    else:    
        br = Browser()    
        r = br.open(request.args.get('urlsearch'))

        if r.code == 200:
            return br.links()
        else:
            return "Error loading page"
7
  • can i have the same result with this ? i just put the beautiful code you gave to me on my Python ex(so Flask),nothing seem change,and i have install beautiful soup already.Should i put this under an @app.route?
    – Nardrek
    Aug 9, 2013 at 8:02
  • can you modify my existing code to see how to do that than we the user put an url on my form input it do like i wanted before ?this is my personal task i want to try to do.i'm quite confuse now.maybe because i start coding 3 month ago :( of course i want to understand first.
    – Nardrek
    Aug 9, 2013 at 8:21
  • i will try this thanks ton1c :) will be right back if issues .
    – Nardrek
    Aug 9, 2013 at 9:17
  • I debug,it works,but same result like before,i run the server,put a URL,and it just open a new page on the browsers and put anything i put before in the search,like "d" see " So when you input a search it go to a new page and show the result ( for exemple:ddddd)" in my first post
    – Nardrek
    Aug 13, 2013 at 2:43
  • What are returning, or what are you printing out on the page?
    – 4d4c
    Aug 13, 2013 at 4:24
1

In Flask you have a map of all urls on app object, you can access it like so:

urlmap = app.url_map.__dict__['_rules_by_endpoint']

so if you want to get all url rules on your page you can add the following at the bottom of your file with app.routes:

def getUrls():
    urlmap = app.url_map.__dict__['_rules_by_endpoint']
    rules = []
    for rule in urlmap:
        rules.append(str(urlmap[rule]))
    result = []
    for rule in rules:
        result.append(rule.split()[1].replace('\'',''))
    return result

Now you can only edit your views file, the one that handles '/search' route:

    if request.method == 'POST':
        urls = getUrls()
        query = str(request.form['urlsearch'])
        if query in urls:
            return str(urls)
        else:
            return "url not in urls"

You also need to edit your html form which at the moment has a bug, you specify form method as get, this needs to be post, otherwise you'll never get to the block which handles post requests:

<form method="post" action="/search">

and voila you'll get a list of ulrs on the page:

['/', '/search', '/static/']
2
  • This is not what he is asking. Aug 9, 2013 at 8:12
  • I tried,Get always a "url not in urls" When putting anything.
    – Nardrek
    Aug 13, 2013 at 3:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.