Hey everyone, I’m stuck with a web scraping project using Selenium and Python. My code runs without any errors, but I’m not getting any output. I’m trying to scrape product info from an e-commerce site and save it as JSON.
I’ve set up Chrome WebDriver (v4.8.2) and it opens the browser, but nothing happens after that. I’ve added time.sleep() to wait for the page to load and I’m using XPaths to find elements. I’ve also tried using print statements to debug, but still no luck.
Here’s a simplified version of my code:
from selenium import webdriver
from selenium.webdriver.common.by import By
import json
import time
driver = webdriver.Chrome()
driver.get('https://example-ecommerce.com/products')
time.sleep(10)
items = driver.find_elements(By.XPATH, '//div[@class="product-card"]')
product_data = []
for item in items:
try:
name = item.find_element(By.XPATH, './/h2[@class="product-name"]').text
price = item.find_element(By.XPATH, './/span[@class="price"]').text
product_data.append({'name': name, 'price': price})
except:
pass
with open('products.json', 'w') as f:
json.dump(product_data, f)
driver.quit()
Any ideas on what I might be doing wrong? Thanks in advance for your help!