Click on a "Show More" button to reveal data

Often, a website will have a Show More button at the bottom of its list of results, which loads extra items onto the page if you click on it.

This tutorial will show you what to do if you need to scrape items that only show up after you click on the Show More button some number of times.

I'll be using this website as an example, so you can follow along.

Clicking on a "show more" button

1. In a new project, before you start selecting items from the page, click the plus button to the right of the select page command, and add a Loop command (under Advanced).

2. Set the options of the Loop command to "for each item in $createArray(20)". You can set the '10' to however many times you need to click "Show More" to get all of the data you need.

For instance, if there are 20 items loaded whenever you click, and you want the first 400 items, you need to click the button 20 times.

3. Click the plus button to the right of the Loop command, and add a Select command.

4. Click on the "Show More" button in Select Mode, to select the button. It will appear in green.

5. Click the plus button to the right of the Select command, and add a Click command.

6. Choose to Continue executing the current template, and press the green button to continue.

7. Now, you can add Select commands from the plus button to the right of the Select page command, to select and extract the data from the items on the page to get product details.

8. With this new select command, click on the first few product names to select all of them on the page. ParseHub will automatically add a Begin New Entry command and Extract commands for the name and url.

Now you can add Relative Select commands to get things such as the product's ratings and prices, or add a Click command to navigate to the product page and scrape the product details

Have more questions? Submit request!

0 Comments

Article is closed for comments.