Hello all, hope we are doing great.
i have built and deployed my app using MUPX to DigitalOcean and it’s working fine. I am also, using PrerenderIO’s to prerender my pages.
But I have one issue. BAIDUSPIDER It keeps crawling my website using my site’s IP address and it’s always having crawl errors. Any Idea on how to stop this crawler? I have used my robots.txt file like this but it keeps crawling my site.
Disallow: /* `