Quantcast
Channel: Active questions tagged selenium - Stack Overflow
Viewing all articles
Browse latest Browse all 98819

Scraping data from a table using BeautifulSoup and Selenium

$
0
0

I am trying to build an application that scrapes course information from a universities course catalogue and then constructs a few possible schedules a student could choose. The course catalogue url doesn't change each time a new course is searched for which is why I am using Selenium to automatically search for a course catalogue then Beautiful Soup to scrape the information. This is my first time using Beautiful Soup and Selenium so apologies in advance if the solution is quite simple.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup
import requests

URL = "http://saasta.byu.edu/noauth/classSchedule/index.php"
driver = webdriver.Safari()
driver.get(URL)
element = driver.find_element_by_id("searchBar")
element.send_keys("C S 142", Keys.RETURN)
response = requests.get(URL);
soup = BeautifulSoup(response.content, 'html.parser')
table = soup.find_all("tbody")
print(table)

Currently, when I print(table) it prints two objects. One, (first picture) has the general information regarding the course (information I don't need to scrape). The second object is empty. As far as I can tell there are only two tables on the website, both pictured below. The second one is the one I am interested scraping but for some reason the second element in table is empty.

enter image description here

The information is pictured below is the information I am trying to scrape.

This is the information I am trying to scrape

Output from the print(table)

<tbody>
   \n
   <tr>
      <th scope="row">Hours</th>
      <td id="courseCredits"></td>
   </tr>
   \n
   <tr>
      <th scope="row">Prerequisites</th>
      <td id="coursePrereqs"></td>
   </tr>
   \n
   <tr>
      <th scope="row">Recommended</th>
      <td id="courseRec"></td>
   </tr>
   \n
   <tr>
      <th scope="row">Offered</th>
      <td id="courseOffered"></td>
   </tr>
   \n
   <tr>
      <th scope="row">Headers</th>
      <td id="courseHeaders"></td>
   </tr>
   \n
   <tr>
      <th scope="row">Note</th>
      <td id="courseNote"></td>
   </tr>
   \n
   <tr>
      <th scope="row">When\xa0Taught</th>
      <td id="courseWhenTaught"></td>
   </tr>
   \n
</tbody>
, 
<tbody></tbody>
]

Viewing all articles
Browse latest Browse all 98819

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>