1

My code is just making a database which stores links. But due to some reason when i open the database, it is an empty table. I have no idea why does this happen.

Not Causing any trouble

#!/usr/bin/python
import urllib2
from BeautifulSoup import *
import sqlite3

count = 0
ecount = 0
aj = list()
error = list()
run = 1

The sql connect

conn = sqlite3.connect('example.db')
cur = conn.cursor()
sql = '''CREATE TABLE URL (LINKS CHAR(512), ID INT(512))'''
cur.execute(sql)

Not Causing any trouble

def retriver(durl):
    newurl = list()
    url = durl
    notpar = {"#", "None", "NONE", "javascript:void(0)"}


    hl = urllib2.urlopen(url, None)
    html = hl.read()
    hl.close()
    soup = BeautifulSoup(html)
    tags = soup('a')

THIS IS THE SCRAPER

    for tag in tags:
        new = tag.get('href', None)
        newurl.append(new)
        for i in newurl:
            try:
                if i[0] == "/" and i[0:3] != "www":
                    if i[len(a)] == "/":
                        i = url[0:len(url)-1] + i
                    else:
                        i = url + i
                if i[0:3] == "www":
                    i = "/" + i
                if "." not in list(i):
                    continue
                if i[0] == "/":
                    i = i[1:len(i)]
                if i[0:4] != "http":
                    if i[len(i)] != "/":
                        i = "/" + i

THIS IS WHERE THE PROBLEM STARTS

                if i not in aj:
                    if i not in notpar:
                        aj.append(i)
                        print i
                        count += 1
                        sql = '''INSERT INTO URL(LINKS, ID) VALUES (%s, %d)'''
                        sqldata = (i, count)
                        cur.execute(sql, sqldata)
                        conn.commit()
            except:
                error.append(i)
                ecount += 1
                global ecount
    global count
    global aj

u = raw_input('Enter the url :: ')
aj.append(u)
retriver(u)

Not Causing any trouble

while True:

    try:
        retriver(aj[run])
        run += 1
        if run > 20:
            break
        print run
    except:
        run += 1
        retriver(aj[run])
        print run

2 Answers 2

1

In this line:

sql = '''INSERT INTO URL(LINKS, ID) VALUES (%s, %d)'''

Try changing it to this:

sql = '''INSERT INTO URL(LINKS, ID) VALUES (?, ?)'''

This post here references that the database's API for parameter substitution uses a ? rather than the %s or %d symbol.

0

I think you should not use query like sql = '''INSERT INTO URL(LINKS, ID) VALUES (%s, %d)'''

For inserting variables directly in your query use format strings like this :

sql = cur.execute("INSERT INTO URL(LINKS, ID) VALUES (?,?)",(x,y))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.