I have a simple model. models.py
class Content(models.Model):
title = models.CharField(max_length = 300, unique = True)
category = models.CharField(max_length = 50)
def __str__(self):
return self.title
then I'm scrapping the data from some sources and want to store in this table.
toi_r = requests.get("some source")
toi_soup = BeautifulSoup(toi_r.content, 'html5lib')
toi_headings = toi_soup.findAll("h2", {"class": "entry-title format-icon"})[0:9]
toi_category = toi_soup.findAll("a", {"class": ""})[0:9]
toi_news = [th.text for th in toi_headings]
toi_cat = [tr.text for tr in toi_category]
for th in toi_headings:
toi_news.append(th.text)
for tr in toi_category:
toi_cat.append(tr.text)
for title, category in zip(toi_news, toi_cat):
n = Content.objects.create(title=title, category=category)
So here, error is UNIQUE Constraint failed. So, If I remove the UNIQUE Constraint, then everything works but mulitple copies of data are stored as the page refreshes. Is there any other way to work with this?
try
/except
block. – Selcuk Sep 30 '20 at 3:00IntegrityError
and continue the loop – Iain Shelvington Sep 30 '20 at 3:07