In a web page that has the Show More button, I click it in a loop till it isnt around anymore (I can see the entire page). Now I need to get some data but the data that I'm getting is the same as before clicking the Show More button.
This is the code that does this:
bodyBefore = response.xpath('/body').get()
# Click the Show More button till it isn't anymore
showmore_btn = self.driver.find_elements_by_xpath(
"//a[@class='event__more event__more--static']")
while len(showmore_btn) > 0:
showmore_btn[0].send_keys(Keys.ENTER)
# Add more time if the previous command doens`t work (Bad internet connection)
time.sleep(5)
showmore_btn = self.driver.find_elements_by_xpath(
"//a[@class='event__more event__more--static']")
bodyAfter = response.xpath('/body').get()
I can't get the new html code in order to scrape it. (With bodyBefore and bodyAfter I can prove this easily)
Does someone know how to do this?