Quantcast
Channel: Active questions tagged selenium - Stack Overflow
Viewing all articles
Browse latest Browse all 97808

Extracting Table data using Selenium and Python into pandas dataframe

$
0
0

so I have done data extract from a table using library BeautifulSoup with code below:

        if soup.find("table", {"class":"a-keyvalue prodDetTable"}) is not None:
        table = parse_table(soup.find("table", {"class":"a-keyvalue prodDetTable"}))
        df = pd.DataFrame(table)

So this worked, I get the table nad parse it out into datafram, however i am trying to do something similar on different website using selenium and here is my code so far:

driver = webdriver.Chrome()
i = "DCD710S2"
base_url = str("https://www.lowes.com/search?searchTerm=" + str(i))
driver.get(base_url)
table = driver.find_element_by_xpath("//*[@id='collapseSpecs']/div/div/div[1]/table/tbody")

So I am getting to the table and i tried to use getAttribute(innerHTML) and some other getAttribute elements but i am unable to get the table as is into pandas. Any suggestion how to handle that with selenium?

Here is how html looks: enter image description here


Viewing all articles
Browse latest Browse all 97808

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>