Some websites, especially e-Commerce websites, have elements that change based on certain parameters. For example, the price of a product on its details page might change based on the options that you choose. In this tutorial, we are going to show you how to scrape these changing elements (price, in this example).
For this example, we will be scraping the Sole Supremacy website, which is a sneaker buying website. The price displayed on the product pages changes based on what size of shoe you are viewing.
Scraping changing values
1. To begin, click on the plus button nest to "Select page" and add a Select command.
2. Using this Select command, click on the first product's name. It will be highlighted in green once it is selected correctly.
3. Similar elements will be highlighted in yellow. Click on one of these highlights in order to select all of the sneakers on this page. You'll see the number 20 in parentheses denoting how many elements are being selected. You can also rename this command by clicking on it and typing in a new name. I've named the command "sneaker" in this example.
4. Now, we want to click into each listing to scrape the prices for these sneakers and their different sizes. To do this, click on the plus button next to "Select sneaker" and choose a Click command.
5. A configuration pop-up will appear asking you if this is a next page button. Choose "No", and create a new template. I have renamed the new template to "sneaker_details", but you can name it whatever you want. Now, click on the "Create New Template" button at the bottom of the configuration pop-up.
6. Using the Select command that is automatically created for you, click on the sizes' dropdown menu. Another Select command will automatically be generated, with which you should select the first size that appears. Click on another size highlighted in yellow so that all of the sizes in the dropdown menu are selected. Rename your commands if you wish - I have named them "dropdown" and "size"
7. To tell ParseHub to click on each sneaker size, click on the plus button next to "Select size" and choose a click command.
8. Like before, respond "No" to whether or not we are clicking on a next page button and Continue executing the current template.
9. Now, you can add commands to Select and Extract your changing value. Click on the plus button next to "Select size", click on "Advanced", and choose a Select command.
10. Select the price of the sneaker by clicking on it, and it will automatically be extracted for you as well. Rename your new Select command to "price".
That's it! The project that we've set up will scrape the prices for all sizes of the sneakers on the first page. If you would like to set up your project to scrape data from all pages, check out this tutorial about pagination.