Quantcast
Channel: Active questions tagged selenium - Stack Overflow
Viewing all articles
Browse latest Browse all 97826

Making changes with Selenium in a web page and the response of the driver returns the same value (Scrapy, Python)

$
0
0

In a web page that has the Show More button, I click it in a loop till it isnt around anymore (I can see the entire page). Now I need to get some data but the data that I'm getting is the same as before clicking the Show More button.

This is the code that does this:

    bodyBefore = response.xpath('/body').get()

    # Click the Show More button till it isn't anymore
    showmore_btn = self.driver.find_elements_by_xpath(
        "//a[@class='event__more event__more--static']")

    while len(showmore_btn) > 0:
        showmore_btn[0].send_keys(Keys.ENTER)
        # Add more time if the previous command doens`t work (Bad internet connection)
        time.sleep(5)
        showmore_btn = self.driver.find_elements_by_xpath(
            "//a[@class='event__more event__more--static']")

    bodyAfter = response.xpath('/body').get()

I can't get the new html code in order to scrape it. (With bodyBefore and bodyAfter I can prove this easily)

Does someone know how to do this?


Viewing all articles
Browse latest Browse all 97826

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>