I'm working on a web scraping script using SeleniumBase on an Ubuntu virtual machine (VM), and I'm running into issues with headless mode and xvfb. This script works perfectly on my local Windows machine, but when I run it on Ubuntu, it gets stuck during execution, specifically at the point where it should interact with elements on the Servipag site.
Here’s the code that works locally on Windows but fails on the Ubuntu VM:
def scrape_servipag_service_reading(service_type, company, identifier, companyType=None, propertyId=None):max_retries = 5result = Nonefor attempt in range(max_retries): with SB(uc=True, xvfb=True) as sb: try: user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" sb.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": user_agent}) sb.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})") sb.set_window_size(1920, 1000) url = f"https://portal.servipag.com/paymentexpress/category/{service_type}" sb.uc_open_with_reconnect(url, 4) sb.sleep(2) # Additional code to handle interactions with Servipag except Exception as e: logger.error(f"Attempt {attempt + 1}/{max_retries} failed with error: {e}") finally: if result is not None: breakreturn result
What I’ve tried in the VM:
Running in headless mode (the driver doesn’t run).Using xvfb=True as part of the SeleniumBase initialization.Verifying that it’s not an element locator issue, as the same code works flawlessly on Windows.On Ubuntu, it seems the driver fails to launch correctly or gets stuck when opening the URL. I'm using the uc=True (undetected Chrome) setting and specifying a user_agent to avoid detection, but no luck so far.
Environment:
OS: Ubuntu (in a virtual machine)
Chrome version : 129.0.6668.101 (Tried both headless and with xvfb)
SeleniumBase version: 4.32.9
Python version: 3.10.0
Question:
Has anyone experienced similar issues with SeleniumBase and xvfb on Ubuntu? Are there additional configurations required for xvfb or for undetected Chrome mode to work on an Ubuntu VM?
Any suggestions or advice would be greatly appreciated!