Does ParseHub respect robots.txt?

Follow