I'm trying to extract HTML data from a table, which consists of about a thousand records, and each records only show 10 of them, thus there are many pages of that table. I have successfully combined Selenium and BeautifulSoup to loop through all the pages and extract the HTML data with for loop, however, when I tried to scrape for five pages of the table, the results shows the HTML data from the first page, being scraped 5 times, although Selenium did redirect the table to page 6.
Scenario:
I am trying to scrape NBA stars' names off the HTML table from the NBA site https://www.nba.com/players
. It can be seen that there are 50 players on each page of the table. I want to scrape 100 names of NBA stars, which would require me to scrape the first 2 pages of the table.
My code is as below:
url = 'https://www.nba.com/players'
driver.get(url)
page = requests.get(url)
print(page)
soup = BeautifulSoup(page.text, 'html.parser')
print(soup)
table = soup.table
table = soup.find(class_='players-list')
names[]
for pages in range(2):
for names in table.find_all('tbody'):
name_rows = names.find_all('tr')
for name_row in name_rows:
player_names = name_row.find('td', class_ = 'primary text RosterRow_primaryCol__1lto4').text
names.append(player_names)
But the output I received contains the players' names on the first page scraped twice:
['PreciousAchiuwa',
'StevenAdams',
'BamAdebayo',
'OchaiAgbaji',
'SantiAldama',
'NickeilAlexander-Walker',
'GraysonAllen',
'JarrettAllen',
'JoseAlvarado',
'KyleAnderson',
'GiannisAntetokounmpo',
'KostasAntetokounmpo',
'ThanasisAntetokounmpo',
'ColeAnthony',
'O.G.Anunoby',
'RyanArcidiacono',
'DeniAvdija',
'DeandreAyton',
'UdokaAzubuike',
'IbouBadji',
'MarvinBagley III',
'PatrickBaldwin Jr.',
'LaMeloBall',
'LonzoBall',
'MoBamba',
'PaoloBanchero',
'DesmondBane',
'DalanoBanton',
'DominickBarlow',
'HarrisonBarnes',
'ScottieBarnes',
'RJBarrett',
'WillBarton',
'CharlesBassey',
'KeitaBates-Diop',
'NicolasBatum',
'DariusBazley',
'BradleyBeal',
'MalikBeasley',
'MarJonBeauchamp',
'DavisBertans',
'PatrickBeverley',
'SaddiqBey',
'KhemBirch',
'GogaBitadze',
'BismackBiyombo',
'BuddyBoeheim',
'BogdanBogdanovic',
'BojanBogdanovic',
'BolBol',
'PreciousAchiuwa',
'StevenAdams',
'BamAdebayo',
'OchaiAgbaji',
'SantiAldama',
'NickeilAlexander-Walker',
'GraysonAllen',
'JarrettAllen',
'JoseAlvarado',
'KyleAnderson',
'GiannisAntetokounmpo',
'KostasAntetokounmpo',
'ThanasisAntetokounmpo',
'ColeAnthony',
'O.G.Anunoby',
'RyanArcidiacono',
'DeniAvdija',
'DeandreAyton',
'UdokaAzubuike',
'IbouBadji',
'MarvinBagley III',
'PatrickBaldwin Jr.',
'LaMeloBall',
'LonzoBall',
'MoBamba',
'PaoloBanchero',
'DesmondBane',
'DalanoBanton',
'DominickBarlow',
'HarrisonBarnes',
'ScottieBarnes',
'RJBarrett',
'WillBarton',
'CharlesBassey',
'KeitaBates-Diop',
'NicolasBatum',
'DariusBazley',
'BradleyBeal',
'MalikBeasley',
'MarJonBeauchamp',
'DavisBertans',
'PatrickBeverley',
'SaddiqBey',
'KhemBirch',
'GogaBitadze',
'BismackBiyombo',
'BuddyBoeheim',
'BogdanBogdanovic',
'BojanBogdanovic',
'BolBol'
]