Quantcast
Channel: Active questions tagged selenium - Stack Overflow
Viewing all articles
Browse latest Browse all 98825

Python Selenium : Why url and page source don't change when in headless mode

$
0
0

I am writing some scrapping script in python and selenium. It works very well in non-headless mode in both chrome and firefox. Now, on switching to headless mode, strange things appear. My code first signings in then go to a link of the thing I want to scrap.

In headless mode, the url after signing in doesn't change on my demand. The HTML source doesn't change to the one of url I demanded, leading to not finding the elements. On my windows machine (win 8). This behavior happens in headless chrome but not in headless firefox. Now, deploying my code into ubuntu server (I only have terminal ssh So, I must use headless mode). Neither of them works. I tried to make every single solution I found, but nothing works.

the relevant part of my code (assuming we are inside class method):

self.driver.get(self.link)
#Some code regards signing in,username,password,..etc and those work fine
if self.login==True:
    self.driver.get(self.link)
    #when the above didn't work, I tried executing javascript code to force it to move to the new url
    #but no success
    #self.driver.execute_script("document.location.assign('https:mylink.com/...')" )

    #Finding elements by xsspath which is giving me timeout exception
    logging.debug("Clicking dropdowns")
            drops =  WebDriverWait(self.driver, 60).until(         
    EC.presence_of_all_elements_located((By.XPATH,"//i[contains(@class,'a-icon a-icon-section-expand')]" )) )
            for drop in drops:
                drop.click()
                logging.debug("Dropdown clicked")

I tried to wait for the url to be changed, but no success. I traced back the problem and found that the url I am in after sign in is just another one to enter the password again (don't know why) so I just located elements again and entered the password again, but problem still appears, I don't move to the next page even by force.

Hope anyone have solution for this.


Viewing all articles
Browse latest Browse all 98825

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>