Selenium web scraping not producing results for e-commerce site

Ecom
1

Hey everyone, I’m stuck with a web scraping issue. I’m trying to grab data from an online store using Selenium and Python, but I’m not getting any output. The weird thing is, there are no error messages either. My code runs fine, and the Chrome window opens up, but nothing happens after that.

Here’s what I’ve tried:

from selenium import webdriver
from selenium.webdriver.common.by import By
import json
import time

browser = webdriver.Chrome()
browser.minimize_window()

store_url = "https://example-store.com/products/12345"
browser.get(store_url)
time.sleep(10)

item_containers = browser.find_elements(By.XPATH, "//div[@class='product-card']")

item_data = []
for container in item_containers:
    try:
        name = container.find_element(By.XPATH, ".//h3[@class='product-name']").text
        cost = container.find_element(By.XPATH, ".//span[@class='product-price']").text
        stars = container.find_element(By.XPATH, ".//div[@class='product-rating']").text
        item_data.append({'name': name, 'cost': cost, 'stars': stars})
    except:
        pass

with open('store_data.json', 'w') as f:
    json.dump(item_data, f)

browser.quit()

I’ve even tried adding print statements, but nothing shows up. Any ideas on what might be going wrong? I’m using Selenium 4.8.2 if that helps. Thanks in advance!

2

I’ve encountered similar issues before. One thing to consider is whether the site is using JavaScript to render content. Selenium might be loading the page before the dynamic content appears. Try increasing the wait time or implementing explicit waits for specific elements. Another possibility is that the site has anti-scraping measures in place. Some e-commerce platforms detect automated browsers and block or redirect them. You might need to add headers or use a more sophisticated approach to mimic human browsing behavior. Have you verified the XPaths are correct? Sometimes site structures change, breaking previously working selectors. Use your browser’s developer tools to ensure the XPaths still match the elements you’re targeting. Lastly, consider adding some logging or debug statements throughout your code to pinpoint where exactly the process is failing. This can provide valuable insights into what’s happening behind the scenes.

3

yo, i had a similar issue. try using a headless browser instead of opening chrome. it might be that the site is detecting selenium n blocking it. also, check ur network connection - sometimes that can mess things up. oh, and make sure ur chromedriver is up to date with ur chrome version. those things helped me when i was stuck

4

Hey SailingBreeze! I can totally relate to that frustrating feeling when your code runs but doesn’t do what you expect. :thinking: Have you considered that the site might be using dynamic content loading or AJAX? Sometimes that can throw a wrench in our scraping plans!

Maybe the site needs more time to load? You could increase that sleep time or, even better, use WebDriverWait for specific elements.

What about checking if there’s an iframe? Some sites load products in those.

Also, have you tried printing out the page source? It might give you clues about what’s actually there when Selenium sees the page.

And regarding those XPATHs, are you sure they’re correct for this particular site? Sometimes I find it helpful to use the browser’s inspect tool to double-check.

What do you think? Do any of these suggestions spark an idea for your next steps?