Meteor filtering out robots, preventing crawling

fuzzywalrus · May 14, 2019, 4:12pm

Currently I’ve been battling the uncrawlable site. I have prerender.io working, and pages are caching on a Heroku hosted Meteor React app. There’s a sitemap.xml, I have metadata for the usual suspects like description, OG tags, Twitter Cards, there isn’t a robots.txt nor any noindex tags.

It’s on 1.4.4.2, a bit older but seems to be fine in every other regard.

doctorpangloss · May 16, 2019, 8:10am

Try running it locally and doing a CURL request that’s identical to a bot’s.

Either you misconfigured prerender.io or your HTTPS is misconfigured.

mikeacre · May 18, 2019, 2:40pm

In order to use prerender properly you need to tell your webserver to route to the prerender server instead of your web app if its a bot. I based my logic around this NGINX config file.

gist.github.com

https://gist.github.com/thoop/8165802

nginx.conf

# Change YOUR_TOKEN to your prerender token
# Change example.com (server_name) to your website url
# Change /path/to/your/root to the correct value

server {
    listen 80;
    server_name example.com;
 
    root   /path/to/your/root;
    index  index.html;

This file has been truncated. show original