Help needed: Selenium Python scraping not producing results

Ecom
1

Hey everyone, I’m stuck with a web scraping project using Selenium and Python. I’m trying to get data from an e-commerce site and save it as JSON. My code runs without any errors, but I’m not getting any output. I’ve even tried using print statements, but nothing shows up.

Here’s a simplified version of what I’m working with:

from selenium import webdriver
from selenium.webdriver.common.by import By
import json
import time

driver = webdriver.Chrome()
driver.get("https://example-shop.com/products")
time.sleep(10)

product_cards = driver.find_elements(By.CLASS_NAME, "product-card")

product_info = []
for card in product_cards:
    try:
        title = card.find_element(By.CLASS_NAME, "product-title").text
        price = card.find_element(By.CLASS_NAME, "product-price").text
        rating = card.find_element(By.CLASS_NAME, "product-rating").text
        product_info.append({"title": title, "price": price, "rating": rating})
    except:
        pass

with open('data.json', 'w') as f:
    json.dump(product_info, f)

driver.quit()

I’m using Selenium 4.8.2 and the Chrome browser opens, but I still get no results. Any ideas what might be going wrong? I’ve tried solutions from other posts but no luck so far.

2

I’ve encountered similar issues in my web scraping projects. One potential problem could be the site’s loading mechanism. Many modern e-commerce platforms use lazy loading or infinite scroll, which can trick Selenium into thinking the page is fully loaded when it’s not.

Try implementing a scroll function in your script. Something like this might help:

 def scroll_to_bottom(driver):
     last_height = driver.execute_script("return document.body.scrollHeight")
     while True:
         driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
         time.sleep(2)
         new_height = driver.execute_script("return document.body.scrollHeight")
         if new_height == last_height:
             break
         last_height = new_height

Call this function before you start scraping. It should force the page to load all products. Also, consider using explicit waits instead of time.sleep() for more reliable element detection. If these don’t work, the site might be using advanced anti-scraping techniques, and you may need to explore alternative methods.

3

Yo, WittyCodr99! Sounds like a real head-scratcher. Have u tried checking the site’s robots.txt file? Some sites block scraping attempts. Also, maybe the site uses AJAX to load stuff. Try waiting for specific elements to appear instead of using time.sleep(). And double-check ur XPath/CSS selectors - they mite be off. Good luck, mate!

4

Hey there WittyCodr99! Interesting problem you’ve got there. I’ve run into similar issues before, and it can be super frustrating when everything seems right but nothing’s happening.

Have you considered that maybe the site is using JavaScript to load content dynamically? Sometimes that can trip up Selenium. You might want to try adding a longer wait time or using WebDriverWait instead of time.sleep().

Also, are you sure the class names you’re using are correct? Websites sometimes use different class names for mobile vs desktop views. Maybe try inspecting the page source to double-check?

Oh, and one more thing - have you tried printing the length of product_cards after you fetch them? That might give you a clue if Selenium is finding the elements at all.

What kind of e-commerce site is it, by the way? Some sites have pretty robust anti-scraping measures. Ever thought about using their API instead, if they have one?

Keep us posted on what you find out! Scraping can be a real puzzle sometimes, but that’s what makes it fun, right?