I'm trying to build a data scraping tool, however, the values of the table change after applying a necessary filter. I'm not sure how to apply the filter using selenium or other tools.
My plan was to load the base table and then figure out how to apply the filter and retrofit my code but I am still stuck even getting the base table off the webpage. The filter I'm am trying to apply is located on a dropdown tool bar labeled "Slates" on the site 'https://rotogrinders.com/projected-stats/nfl'
I'm fairly confident this code is getting the correct table:
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
url = 'https://rotogrinders.com/projected-stats/nfl-qb?site=fanduel'
driver.get(url)
table = driver.find_element_by_xpath("//*[@id='proj-stats']")
However converting it into a pandas dataframe hasn't been going well.
results_table = []
for row in table:
temp = []
columns = row.find_element_by_xpath("//*[@id='proj-stats']/div[1]")
for column in columns:
temp.append(column.text)
results_table.append(temp)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-79-bdda19bc35a3> in <module>
1 results_table = []
----> 2 for row in table:
3 temp = []
4 columns = row.find_element_by_xpath("//*[@id='proj-stats']/div[1]")
5 for column in columns:
TypeError: 'WebElement' object is not iterable