Quantcast
Channel: Active questions tagged selenium - Stack Overflow
Viewing all articles
Browse latest Browse all 98788

How to loop from a list of urls by clicking the xpath and extract data using Selenium in Python?

$
0
0

I am extracting board members from a list of URLs. For each url in the URL_lst, click the first xpath (ViewMore to expand the list), then extract values from the second xpath (BoardMembers' info). Below are the three companies I want to extract info: https://www.bloomberg.com/quote/FB:US, https://www.bloomberg.com/quote/AAPL:US, https://www.bloomberg.com/quote/MSFT:US

enter image description here

My code is shown below but doesn't work. The Output list is not aggregated. I know sth wrong with the loop but don't know how to fix it. Can anyone tell me how to correct the code? Thanks!

URL_lst = ['https://www.bloomberg.com/quote/FB:US','https://www.bloomberg.com/quote/AAPL:US','https://www.bloomberg.com/quote/MSFT:US']

Outputs = []
driver = webdriver.Chrome(r'xxx\chromedriver.exe')

for url in URL_lst:
    driver.get(url)
    for c in driver.find_elements_by_xpath("//*[@id='root']/div/div/section[3]/div[10]/div[2]/div/span[1]"):
        c.click()
        for e in (c.find_elements_by_xpath('//*[@id="root"]/div/div/section[3]/div[10]/div[1]/div[2]/div/div[2]')[0].text.split('\n'):
            Outputs.append(e)

print(Outputs)

Viewing all articles
Browse latest Browse all 98788

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>