Prevent infinite loops

Many websites display results on more than one page. After selecting all the results on the first page, you should navigate to the next pages of the results to extract information from all the results. You can follow the "Click on the "Next" button to scrape multiple pages (pagination)" tutorial to do so.

When selecting the "next" button, you want to make sure that you did not create an infinite loop in the project.

How to prevent an infinite loop

On some websites the "next" button is still visible on the last page of the results, although it is disabled and not click-able.

First, we need to make sure that the "next" button is not available on the last page: switch to the Browse Mode on ParseHub and go to the last page of the results.

Click on the "Select button" node and make sure that the selection node shows (0). If it is shows (1), this means that we are still selecting the "next" button on the last page. You can also see that the next button is visible (normally faded) on the webpage.
 
In this case your project is selecting a disabled button, and you are clicking on it infinitely. To prevent creating this infinite loop, you should add a conditional command to skip the "next" button if it is disabled.
 
You should add the conditional node right after the Select button node and enter !$e.prop("class").toLowerCase().contains("disabled") in the conditional text box, without the surrounding quotations.
 

If this doesn't work

It could be that the element isn't using a "class" tag to check if it's disabled.
 
If you know XPath or CSS, or read our tutorial on XPath, you can look into the HTML of the page and try to resolve the issue manually.
 
Or, send us a message and we'll help you to get the data you need as soon as we can.

Have more questions? Submit request!

0 Comments

Article is closed for comments.