I have a code here that scrapes data from a specific website which is https://vancouver.craigslist.org/search/ela
. My problem is when I execute my code, it gives me an error of 'list' object has no attribute 'get_attribute'
in the line asdf = images.get_attribute("src")
. I am using selenium library in scraping the data. What I want is to insert the image url from my table which is named images
but I cannot. What is wrong with my code? I am not familiar with python yet thats why I am asking questions. Thanks a lot for consideration.
Current code
x = driver.find_elements_by_class_name('hdrlnk')
y = driver.find_elements_by_xpath('//p[@class="result-info"]/span[@class="result-meta"]//span[@class="result-price"]')
images = driver.find_elements_by_xpath('//*[@id="sortable-results"]/ul/li/a/img')
for img in images:
print(img.get_attribute('src'))
for i in range(len(x)):
asdf = images.get_attribute("src")
prod = (x[i].text)
price = (y[i].text)
image = asdf
sql = """INSERT INTO products (name,price,image) VALUES (%s,%s,%s)"""
mycursor.execute(sql,(prod,price,image))
mydb.commit()
When I comment this line
for img in images:
print(img.get_attribute('src'))
and remove the asdf
and image
variable, I am able to insert the data and also when I comment this line of code and remain the print for images,
#for i in range(len(x)):
#asdf = images.get_attribute("src")
#prod = (x[i].text)
#price = (y[i].text)
#image = asdf
#sql = """INSERT INTO products (name,price,image) VALUES (%s,%s,%s)"""
#mycursor.execute(sql,(prod,price,image))
#mydb.commit()
I got the result I want which is like this
https://images.craigslist.org/00z0z_4cqgwC5PIXs_300x300.jpg
https://images.craigslist.org/00J0J_f6AnAonGjXd_300x300.jpg
https://images.craigslist.org/00606_mtKNjKREOO_300x300.jpg
https://images.craigslist.org/00U0U_l5t0QnjZEPt_300x300.jpg
https://images.craigslist.org/00505_gIXt1C8aeqk_300x300.jpg
https://images.craigslist.org/00N0N_6P1GmSiL2vI_300x300.jpg
Sample data for x
and y
variable in i
loop:
x = Spigen Magnetic Car Phone Mount
y= $20
What do I need to do in order to insert the image url with the product name and images in a single row? TIA.
EDIT. I tried @terahertz's answer and rewrite my code like this
x = driver.find_elements_by_class_name('hdrlnk')
y = driver.find_elements_by_xpath('//p[@class="result-info"]/span[@class="result-meta"]//span[@class="result-price"]')
images = driver.find_elements_by_xpath('//*[@id="sortable-results"]/ul/li/a/img')
for img in images:
# print(img.get_attribute('src'))
for i in range(len(x)):
asdf = img.get_attribute("src")
prod = (x[i].text)
price = (y[i].text)
image = asdf
sql = """INSERT INTO products (name,price,image) VALUES (%s,%s,%s)"""
mycursor.execute(sql,(prod,price,image))
mydb.commit()
Current DB datas
+-----+------------------------------------------------------------------------+--------+-------------------------------------------------------------+
| id | name | price | image |
+-----+------------------------------------------------------------------------+--------+-------------------------------------------------------------+
| 1 | Spigen Magnetic Car Phone Mount | $20 | https://images.craigslist.org/00i0i_7PvHxDMvR2o_300x300.jpg |
| 2 | Netgear Nighthawk x6 r8000 wireless router | $120 | https://images.craigslist.org/00i0i_7PvHxDMvR2o_300x300.jpg |
| 3 | iPod Touch 8gb 2nd generation - Loaded with Classic Rock | $60 | https://images.craigslist.org/00i0i_7PvHxDMvR2o_300x300.jpg |
| 4 | 3 plug 3.1A fast USB wallplugs | $10 | https://images.craigslist.org/00i0i_7PvHxDMvR2o_300x300.jpg |
| 5 | Audio and Video Cables | $3 | https://images.craigslist.org/00i0i_7PvHxDMvR2o_300x300.jpg |
| 6 | Like New Samsung 50" HD TV ForSale | $400 | https://images.craigslist.org/00i0i_7PvHxDMvR2o_300x300.jpg |
| 7 | SONY Alarm Clock | $20 | https://images.craigslist.org/00i0i_7PvHxDMvR2o_300x300.jpg |
| 8 | Bowers & Wilkins P7 Wireless MINT | $450 | https://images.craigslist.org/00i0i_7PvHxDMvR2o_300x300.jpg |
+-----+------------------------------------------------------------------------+--------+-------------------------------------------------------------+
Now I can insert into my database, BUT the problem is the image
column has the same value with other. Its like only one image url inserted. And when I visited the link, the product name and the image doesnt match.