User Agents are being filtered preventing bots like Googlebot, BingBot, Prerender.io and I have no idea how or why

#1

I inherited a React Meteor project running 1.4.4.2 running in Heroku. I set up Meteor with Prerender.io rather easily, authorized my own token using this as my guide https://galaxy-guide.meteor.com/seo.html

I’ve managed to stump Prerender io and Heroku hasn’t been any help. User Agents for for popular crawlers hit the project but then timeout with a 504 error. The crazy part is you can hit the Herokuapp URL if requesting the page as a Google bot or Bingbot, and Prerender io. I’m new to Meteor and for the most part, it’s been fairly straightforward but this particular issue has the previous developer, my team members, Prerender io and of course, me stumped.

To make things more interesting, if I curl as a made up user agent, connections are able to be established, so the filtering appears to be like a blacklist. Grepping the code base for these UAs is a bust.

I’ve relaxed the content-security-policy settings, and nothing changed.

Here’s a list of packages,

meteor-base@1.0.4 # Packages every Meteor app needs to have
mobile-experience@1.0.4 # Packages for a great mobile UX
mongo@1.1.17 # The database Meteor supports right now
session@1.1.7 # Client-side reactive dictionary for your app
jquery@1.11.10 # Helpful client-side library
tracker@1.1.3 # Meteor’s client-side reactive programming library
es5-shim@4.6.15 # ECMAScript 5 compatibility for older browsers.
ecmascript@0.7.3 # Enable ECMAScript2015+ syntax in app code
momentjs:moment
reactive-var@1.0.11
aldeed:simple-schema
standard-minifier-js@2.0.0
react-meteor-data
static-html
blaze
reactive-dict@1.1.8
accounts-password@1.3.6
check@1.2.5
underscorestring:underscore.string
aldeed:collection2
alanning:roles
seba:minifiers-autoprefixer
fourseven:scss
force-ssl@1.0.14
shell-server@0.2.3
okgrow:analytics@3.0.5
http@1.2.12
percolate:synced-cron
browser-policy
mixmax:smart-disconnect
meteorhacks:unblock
themeteorchef:bert
mdg:seo