Python Selenium web scraping not producing output for e-commerce site

Ecom
1

Hey folks, I’m stuck with a web scraping project using Python and Selenium. I’m trying to extract product data from an online store, but my code finishes without errors and still produces no output. I’ve verified that my Selenium version (4.8.2) and Chrome driver are current. I’ve even attempted to use print statements and dump the data into a JSON file, yet nothing appears. Here’s a simplified version of my approach:

from selenium import webdriver
from selenium.webdriver.common.by import By
import json
import time

browser = webdriver.Chrome()
browser.get('https://example-shop.com/products')
time.sleep(10)

product_containers = browser.find_elements(By.CLASS_NAME, 'product-card')

items = []
for container in product_containers:
    try:
        name = container.find_element(By.CLASS_NAME, 'product-name').text
        cost = container.find_element(By.CLASS_NAME, 'product-price').text
        stars = container.find_element(By.CLASS_NAME, 'product-rating').text
        items.append({'name': name, 'cost': cost, 'stars': stars})
    except:
        pass

with open('shop_data.json', 'w') as f:
    json.dump(items, f)

browser.quit()

Any insight into what might be going wrong would be appreciated!

2

Hey there, WittyCodr99! I’m intrigued by your web scraping project. Have you considered that the site might be using dynamic content loading? Sometimes e-commerce sites load products as you scroll, which could explain why you’re not getting any output.

A couple things you might want to try:

  1. Have you checked if the page is fully loaded before scraping? Maybe increase your sleep time or use WebDriverWait instead?

  2. What about the site’s structure? Are you sure the class names you’re using are correct? They might be different or nested differently than you expect.

  3. Have you tried printing out the HTML of the page to see what Selenium is actually seeing?

I’m curious, what made you choose this particular e-commerce site? Is it for a personal project or something else? It’d be cool to hear more about what you’re working on!

3

yo, have u checked if the site uses lazy loading? that could b why ur not gettin any data. try scrolling the page programmatically before scraping:

browser.execute_script(‘window.scrollTo(0, document.body.scrollHeight);’)
time.sleep(2)

also, maybe add some print statements in ur loop to see if its even finding the product containers. good luck with ur project!

4

I’ve encountered similar issues when scraping e-commerce sites. One thing to consider is that many modern websites use JavaScript to render content dynamically, which Selenium might not handle well by default. You could try using a wait condition instead of a fixed sleep time to ensure the page has loaded fully. For example:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(browser, 20)
product_containers = wait.until(EC.presence_of_all_elements_located((By.CLASS_NAME, ‘product-card’)))

Also, check if the site is using any anti-bot measures. Some e-commerce platforms employ techniques to detect and block scraping attempts. You might need to add headers to your requests or use a proxy to bypass these protections. If all else fails, consider using a headless browser or a different scraping library like Scrapy, which might be more effective for certain sites.