I want to extract a table from this website after clicking on "National Data". I know there is an option to download it directly but I'm very interested in web scraping and I want to learn. Previously, I used the Selenium package in Python for automating the process of selecting all the checkboxes containing the years and now I want to extract the table as a dataframe until the last page, in order to have the whole entries in my dataframe. I think the table id I want to scrap is 'DataTables_Table_26'.
I tried to do this for extracting the first table but it doesn't work.
html = driver.page_source
soup = BeautifulSoup(html, "lxml")
table = soup.find_all('table', {'id':'DataTables_Table_26'})
If you need to see the previous lines of my code, just tell me. Thanks in advance.
This is the part of the html code where there is a table object:
