Hey all,
So I have been utilizing Prerender.io for several of my sites, I have it set based on the advised specs form Prerender where my NGINX server determines if it is a bot and sends it to its pre render.
The issue is this is causing a bunch of blank pages to be pre-rendered as it seems random bots keeps trying to access random non existent pages on my site. I’m assuming these are auto bots looking for data or holes in the security, but how do I stop this from happening?
I know there are ways to check if a route exists with SSR but how can I do this with NGINX? One thing I was thinking was only going to pre-render if it was trying to access certain uri’s but that doesn’t seem scallable or efficient.
Has anyone else tackled this?
Mike Acre
Hi Mike!
There are two things you can do here to prevent those pages from being cached in Prerender.io:
-
Use our prerender-status-code meta tag: https://prerender.io/documentation/best-practices. You would basically run your normal javascript routes in the browser, and if the URL ends up being invalid, you would dynamically insert a meta tag with name=“prerender-status-code” and content=“404” into the head of the page. Our Prerender.io browser would see that on the page, return a real 404 status code, and we wouldn’t cache that URL in Prerender.io after seeing the non-200 status code. Since we only cache pages that return a status code of 200, you can return any other status code with the prerender-status-code meta tag and we wouldn’t cache it.
-
If the URLs being accessed only have different query string parameters, you can log into your Prerender.io account and go to the URL Parameters section. From there, you can tell us to ignore certain query string parameters that don’t affect the rendering of the page. That way, we wouldn’t cache URLs with those query strings as separate URLs in Prerender.io.
That should solve things but let me know if you have any other questions!
Todd
1 Like