Infinite Scroll

This tutorial will show you how to get data from web pages that display more results as you scroll. is one example of a website where you only need to scroll, and Twitter is another.

1. If not already in select mode, click on the "Select page" command + button that is located on the right of the command. From the tool menu, choose the "Select" tool.

2. Select the container that holds all of the content that you want to scroll. To do this you need to refine what is highlighted on the page. Hover over any element contained in the scrollable content, such as a product title. Hold down the CTRL key (or the CMD key on a Mac) and press the number 1 on your keyboard. Press 1 again and again until you see a highlight around all of the loaded content. Without letting go of CTRL/CMD, click on the highlight to make a selection. If you zoom too far, press the number 2 on your keyboard to shrink the selection. 

3. The Select & Extract selection1 command should appear for you in the left side bar.

4. Click on the + button next to the selection1 command that selects the entire container. Click on the "Advanced" section to expand the tool menu.

5. From the tool menu choose the "Scroll" tool. 

6. Click on the "align to bottom"' checkbox in the scroll command options.

7. Tell ParseHub how many times you want to scroll down the page. In the first text box enter the number "5" or any other number. If there's more content to load, you'll have to repeat it many times. If needed, you can tell ParseHub to wait longer in between scrolls by adjusting the number in the second text box. 

8. Make the rest of the selections and extractions by clicking on the + button of the "Select page" command. ParseHub will scroll your content and then scrape all of the information from the fully loaded page.

9. To check your results, run your project once on the ParseHub servers, or run it locally with the Test Run debugger. 


Please also see this written tutorial on infinite scrolling with the Publons website as an example.

Have more questions? Submit request!


Article is closed for comments.