Quantcast
Channel: Active questions tagged selenium - Stack Overflow
Viewing all articles
Browse latest Browse all 97771

Python nested loop error when inserting to DB

$
0
0

I have a code here that scrapes data from a specific website which is https://vancouver.craigslist.org/search/ela. My problem is when I execute my code, it gives me an error of 'list' object has no attribute 'get_attribute' in the line asdf = images.get_attribute("src"). I am using selenium library in scraping the data. What I want is to insert the image url from my table which is named images but I cannot. What is wrong with my code? I am not familiar with python yet thats why I am asking questions. Thanks a lot for consideration.

Current code

   x = driver.find_elements_by_class_name('hdrlnk')
    y = driver.find_elements_by_xpath('//p[@class="result-info"]/span[@class="result-meta"]//span[@class="result-price"]')
    images = driver.find_elements_by_xpath('//*[@id="sortable-results"]/ul/li/a/img')
    for img in images:
      print(img.get_attribute('src'))
      for i in range(len(x)):
        asdf = images.get_attribute("src")
        prod = (x[i].text)
        price = (y[i].text)
        image = asdf
        sql =  """INSERT INTO products (name,price,image) VALUES (%s,%s,%s)"""
        mycursor.execute(sql,(prod,price,image))
        mydb.commit()

When I comment this line

for img in images:
      print(img.get_attribute('src'))

and remove the asdf and image variable, I am able to insert the data and also when I comment this line of code and remain the print for images,

   #for i in range(len(x)):
        #asdf = images.get_attribute("src")
        #prod = (x[i].text)
        #price = (y[i].text)
        #image = asdf
        #sql =  """INSERT INTO products (name,price,image) VALUES (%s,%s,%s)"""
        #mycursor.execute(sql,(prod,price,image))
        #mydb.commit()

I got the result I want which is like this

https://images.craigslist.org/00z0z_4cqgwC5PIXs_300x300.jpg
https://images.craigslist.org/00J0J_f6AnAonGjXd_300x300.jpg
https://images.craigslist.org/00606_mtKNjKREOO_300x300.jpg 
https://images.craigslist.org/00U0U_l5t0QnjZEPt_300x300.jpg
https://images.craigslist.org/00505_gIXt1C8aeqk_300x300.jpg
https://images.craigslist.org/00N0N_6P1GmSiL2vI_300x300.jpg

Sample data for x and y variable in i loop:

x = Spigen Magnetic Car Phone Mount
y= $20

What do I need to do in order to insert the image url with the product name and images in a single row? TIA.

EDIT. I tried @terahertz's answer and rewrite my code like this

  x = driver.find_elements_by_class_name('hdrlnk')
    y = driver.find_elements_by_xpath('//p[@class="result-info"]/span[@class="result-meta"]//span[@class="result-price"]')
    images = driver.find_elements_by_xpath('//*[@id="sortable-results"]/ul/li/a/img')
    for img in images:
      # print(img.get_attribute('src'))
      for i in range(len(x)):
        asdf = img.get_attribute("src")
        prod = (x[i].text)
        price = (y[i].text)
        image = asdf
        sql =  """INSERT INTO products (name,price,image) VALUES (%s,%s,%s)"""
        mycursor.execute(sql,(prod,price,image))
        mydb.commit()

Current DB datas

+-----+------------------------------------------------------------------------+--------+-------------------------------------------------------------+
| id  | name                                                                   | price  | image                                                       |
+-----+------------------------------------------------------------------------+--------+-------------------------------------------------------------+
|   1 | Spigen Magnetic Car Phone Mount                                        | $20    | https://images.craigslist.org/00i0i_7PvHxDMvR2o_300x300.jpg |
|   2 | Netgear Nighthawk x6 r8000 wireless router                             | $120   | https://images.craigslist.org/00i0i_7PvHxDMvR2o_300x300.jpg |
|   3 | iPod Touch 8gb 2nd generation - Loaded with Classic Rock               | $60    | https://images.craigslist.org/00i0i_7PvHxDMvR2o_300x300.jpg |
|   4 | 3 plug 3.1A fast USB wallplugs                                         | $10    | https://images.craigslist.org/00i0i_7PvHxDMvR2o_300x300.jpg |
|   5 | Audio and Video Cables                                                 | $3     | https://images.craigslist.org/00i0i_7PvHxDMvR2o_300x300.jpg |
|   6 | Like New Samsung 50" HD TV ForSale                                     | $400   | https://images.craigslist.org/00i0i_7PvHxDMvR2o_300x300.jpg |
|   7 | SONY Alarm Clock                                                       | $20    | https://images.craigslist.org/00i0i_7PvHxDMvR2o_300x300.jpg |
|   8 | Bowers & Wilkins P7 Wireless MINT                                      | $450   | https://images.craigslist.org/00i0i_7PvHxDMvR2o_300x300.jpg |
+-----+------------------------------------------------------------------------+--------+-------------------------------------------------------------+

Now I can insert into my database, BUT the problem is the image column has the same value with other. Its like only one image url inserted. And when I visited the link, the product name and the image doesnt match.


Viewing all articles
Browse latest Browse all 97771

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>