Web scraping e-commerce site with Selenium and Python: No output or errors

Ecom
1

Hey everyone, I’m stuck with a web scraping project. I’m trying to extract data from an online store using Selenium and Python. The weird thing is, my code runs without any errors, but I’m not getting any output. I’ve tried different things:

  • Using Selenium 4.8.2
  • Checking StackOverflow for solutions
  • Adding print statements for debugging

Here’s a simplified version of my code:

from selenium import webdriver
from selenium.webdriver.common.by import By
import json
import time

browser = webdriver.Chrome()
browser.get('https://example-store.com/products')
time.sleep(10)

product_containers = browser.find_elements(By.CLASS_NAME, 'product-card')

item_data = []
for container in product_containers:
    try:
        name = container.find_element(By.CLASS_NAME, 'product-name').text
        cost = container.find_element(By.CLASS_NAME, 'product-price').text
        stars = container.find_element(By.CLASS_NAME, 'product-rating').text
        
        item_data.append({"name": name, "cost": cost, "stars": stars})
    except:
        pass

with open('store_data.json', 'w') as f:
    json.dump(item_data, f)

browser.quit()

The browser window opens, but nothing gets saved to the JSON file. Any ideas what might be going wrong? Thanks in advance for your help!

2

I’ve encountered similar issues in my web scraping projects. One thing to consider is that the website might be using lazy loading or AJAX to populate product data. This can cause Selenium to search for elements before they’re actually loaded.

Try implementing explicit waits instead of using time.sleep(). You can use WebDriverWait to wait for specific elements to be present or visible.

Also, verify if the website employs any anti-bot measures that might block your requests, and double-check your element selectors to ensure they match dynamically generated classes. If these suggestions don’t resolve the issue, consider exploring headless browser options or alternative scraping libraries such as Scrapy or requests-html.

Let me know if any of these approaches help resolve your problem.

3

Hey there WittyCodr99! Interesting problem you’ve got there. I’ve run into similar issues before, and it can be pretty frustrating when everything seems right but nothing’s happening.

Have you considered that the website might be using JavaScript to load the product data? Sometimes, sites load content dynamically, which can throw off our scraping attempts.

A couple things you could try:

  • Wait for elements to be present instead of using a fixed sleep time. Selenium’s WebDriverWait can be really helpful here.
  • Check if there are any iframes on the page. Sometimes content is nested inside these, and you might need to switch to an iframe to access the elements.
  • Print out the page source after the page loads to see if the elements you want are there when Selenium searches for them.
  • Try different locators. If class names have changed or are dynamic, XPath or CSS selectors might be more reliable.

Just throwing some ideas out there! What made you choose Selenium for this project, and have you experimented with any other scraping methods or libraries?

4

hey wittycodr99, i’ve faced similar probs. u might need to scroll or check if elements load dynamically. try implementing scroll actions, use javascript to confirm page readiness, or examine the page source. also, maybe try requests and beautifulsoup instead of selenium. hope this helps!