I am new to Selenium for web-scraping. I have written a Python code which if given the following results page on IMDB:
https://www.imdb.com/search/title/?genres=action&title_type=feature&explore=genres&count=50
I can use Beautiful Soup to parse the html on the page and extract the information I am looking for.
However, I want to automate Selenium to click on the next button each time so I can create a search and navigate to the end.
I have tried using the following code, but I get the error message
browser.find_element_by_css_selector('lister-page-next next-page').click()
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"lister-page-next next-page"} (Session info: headless chrome=77.0.3865.120)
I have also tried
browser.find_element_by_xpath('//a[contains(.,"next-page")]').click()
and also gotten an error.
If someone could point me towards how to get Selenium to click Next, if it exists, that would be very helpful. Thank you.