Web scraping e-commerce site with Selenium and Python: No output despite no errors

Ecom
1

Hey everyone, I’m stuck with a web scraping project. I’m trying to get data from an online store using Selenium and Python. My code runs without any errors, but I’m not getting any output. It’s really frustrating!

I’m using Selenium 4.8.2 and the Chrome webdriver opens up fine. I’ve tried different solutions from Stack Overflow, but nothing seems to work. Here’s what I’m doing:

  1. Opening a product page on the site
  2. Waiting for the page to load
  3. Trying to find product info (title, price, rating) using XPath
  4. Aiming to save the data as JSON

The weird thing is, even when I add print statements, I get nothing. Has anyone run into this before? Any ideas on what I might be missing or doing wrong? I’d really appreciate some help figuring this out!

from selenium import webdriver
from selenium.webdriver.common.by import By
import json
import time

browser = webdriver.Chrome()
browser.minimize_window()

store_url = 'https://example-store.com/product/12345'
browser.get(store_url)
time.sleep(10)

item_containers = browser.find_elements(By.XPATH, '//div[@class="product-card"]')

items_data = []
for container in item_containers:
    try:
        name = container.find_element(By.XPATH, './/h3[@class="item-name"]').text
    except:
        name = 'N/A'
    
    try:
        cost = container.find_element(By.XPATH, './/span[@class="item-price"]').text
    except:
        cost = 'N/A'
    
    try:
        stars = container.find_element(By.XPATH, './/div[@class="item-rating"]').text
    except:
        stars = 'N/A'
    
    items_data.append({
        'name': name,
        'cost': cost,
        'stars': stars
    })

with open('scraped_data.json', 'w') as f:
    json.dump(items_data, f)

browser.quit()

Any help would be awesome!

2

hey mate, i had a similar issue. try using WebDriverWait instead of time.sleep(). it waits for specific elements to load before scraping. Also, check if the XPath is correct - sometimes websites use dynamic class names. and don’t minimize the browser window, it can mess things up. hope this helps!

3

Hey RollingThunder! I totally get your frustration. Web scraping can be tricky sometimes. Have you considered that the site might be using JavaScript to load content dynamically? That could explain why you’re not seeing any errors but still not getting data.

Maybe try using browser.execute_script() to check if the elements are there after the page loads? Something like:

element = browser.execute_script("return document.querySelector('.product-card')")
print(element)

If that returns None, the elements might not be on the page yet. In that case, you might need to interact with the page somehow to trigger the content load.

Also, have you checked the Network tab in Chrome’s dev tools while the page loads? It might give you clues about what’s happening behind the scenes.

Oh, and one more thing - some sites have pretty robust anti-scraping measures. Maybe try adding a user-agent to your requests to make them look more like they’re coming from a real browser?

Let me know if any of that helps or if you need more ideas. Happy to brainstorm more!