Quantcast
Channel: Active questions tagged selenium - Stack Overflow
Viewing all articles
Browse latest Browse all 98509

How to scrap a data table from a webpage after applying a filter

$
0
0

I'm trying to build a data scraping tool, however, the values of the table change after applying a necessary filter. I'm not sure how to apply the filter using selenium or other tools.

My plan was to load the base table and then figure out how to apply the filter and retrofit my code but I am still stuck even getting the base table off the webpage. The filter I'm am trying to apply is located on a dropdown tool bar labeled "Slates" on the site 'https://rotogrinders.com/projected-stats/nfl'

I'm fairly confident this code is getting the correct table:

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
url = 'https://rotogrinders.com/projected-stats/nfl-qb?site=fanduel'
driver.get(url)
table = driver.find_element_by_xpath("//*[@id='proj-stats']")

However converting it into a pandas dataframe hasn't been going well.

results_table = []
for row in table:
    temp = []
    columns = row.find_element_by_xpath("//*[@id='proj-stats']/div[1]")
    for column in columns:
        temp.append(column.text)

    results_table.append(temp)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-79-bdda19bc35a3> in <module>
      1 results_table = []
----> 2 for row in table:
      3     temp = []
      4     columns = row.find_element_by_xpath("//*[@id='proj-stats']/div[1]")
      5     for column in columns:

TypeError: 'WebElement' object is not iterable

Viewing all articles
Browse latest Browse all 98509

Latest Images

Trending Articles



Latest Images

<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>