Getting accurate product rankings from an online store instead of HTML order?

Ecom
1

Hey everyone! I’m stuck on a project where I need to grab the real rankings of items on an online store. Right now, my code is just getting the order from the HTML, which isn’t right.

For instance, it’s showing the 7th best-selling item as 4th because that’s where it is in the HTML. I’ve tried a few things:

  • Looked at HTML tags for ranking info
  • Tried to get ranks from the text on the page
  • Checked out network stuff to see if rankings come from somewhere else

Here’s a bit of my code:

let items = document.querySelectorAll('.item-box');
items.forEach((item, index) => {
  let rankSpot = item.querySelector('.rank-label');
  let trueRank = rankSpot ? rankSpot.textContent : 'Not found';
  console.log(`Item ${index + 1}: Rank ${trueRank}`);
});

I’m wondering:

  1. How do these sites usually handle product rankings?
  2. What’s the best way to get the real order?
  3. Should I be looking at stuff that loads after the page does?

I’m learning web scraping and JavaScript, so any help would be awesome! Thanks!

2

I’ve encountered similar challenges with scraping product rankings. In my experience, many e-commerce sites load their ranking information dynamically via JavaScript after the page initially loads, which means the HTML order does not necessarily represent the true ranking. One method that often works is checking the AJAX requests in the Network tab, as the ranking data might be fetched separately from the page content. Another approach is to inspect the HTML for hidden elements or data attributes that may carry ranking information. If the site offers a public API, accessing it can provide more accurate data. Finally, using a headless browser such as Puppeteer allows all dynamic content to load fully before scraping. Always ensure to respect the website’s robots.txt file and terms of service when performing scraping activities.

3

Hey there DashingDog! Interesting problem you’ve got there. I’m kinda curious - have you tried looking into the site’s JavaScript files? Sometimes the ranking logic is hidden in there. Or maybe the rankings are coming from a database? Could be worth checking if there’s any SQL-like stuff going on behind the scenes.

Oh, and random thought - what if you tried simulating user behavior, like sorting or filtering products? That might trigger the site to reveal its true ranking system.

Anyway, just brainstorming here. What kind of site is this? I’m always fascinated by how different e-commerce platforms handle their product listings. Have you noticed any patterns in how they update their rankings? Like, do they change daily or in real-time?

Keep us posted on what you find out! This stuff is pretty tricky but super interesting to figure out.

4

hey dashinDog, i ran into this too. most stores load rankings dynamicly, so check the network tab for ajax. sometimes hidden html elements or a public api can reveal true ranking. hope this helps!