Selenium web scraping not producing results: Python script runs without errors but no output

Ecom
1

Hey everyone, I’m stuck with a web scraping project using Selenium and Python. My code runs without any errors, but I’m not getting any output. I’m trying to scrape product info from an e-commerce site and save it as JSON.

I’ve set up Chrome WebDriver (v4.8.2) and it opens the browser, but nothing happens after that. I’ve added time.sleep() to wait for the page to load and I’m using XPaths to find elements. I’ve also tried using print statements to debug, but still no luck.

Here’s a simplified version of my code:

from selenium import webdriver
from selenium.webdriver.common.by import By
import json
import time

driver = webdriver.Chrome()
driver.get('https://example-ecommerce.com/products')
time.sleep(10)

items = driver.find_elements(By.XPATH, '//div[@class="product-card"]')
product_data = []

for item in items:
    try:
        name = item.find_element(By.XPATH, './/h2[@class="product-name"]').text
        price = item.find_element(By.XPATH, './/span[@class="price"]').text
        product_data.append({'name': name, 'price': price})
    except:
        pass

with open('products.json', 'w') as f:
    json.dump(product_data, f)

driver.quit()

Any ideas on what I might be doing wrong? Thanks in advance for your help!

2

hey, im wonderin if the elements are loaded fully. try using webdriverwait insted of sleep to let the page update a bit more. also, check the xpaths for change - sometimes the site updates its structure. hope it sortz out!

3

I’ve encountered similar issues before. One thing to consider is whether the website uses dynamic content loading. Some e-commerce sites load products as you scroll, which Selenium might not detect initially. Try scrolling the page before scraping:

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(2)

Also, ensure you’re not hitting any anti-bot measures. Some sites detect automated browsers. You could try adding user-agent headers to mimic a real browser:

options = webdriver.ChromeOptions()
options.add_argument('user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36')
driver = webdriver.Chrome(options=options)

If these don’t work, try printing the page source to see what content is actually being loaded.