What's the current best method to ensuring your site is crawlable by Google

Something that has been plaguing me since I first put out my site is the fact that when I use Google Search Console to preview how Google sees my site I get very inconsistent results.

Sometimes it loads the page correctly. Sometimes it doesn’t load anything. But most commonly it will load the layoutTemplate of iron router, while not rendering the actual main content of the page (If this means anything, it has to retrieve mongo collection fetches before it can load that page, but I don’t think this should matter).

This is now severly impacting my search engine presence. I’m aware of stuff like “prerender” which attemps to take screenshots of your page and serve them to google. But not only is this service costly, it seems really hacky.

Also the fact that I can get Google to successfully load a preview of my webpage SOMETIMES makes me think that I’m close.

1 Like

I am using “prerender.io” as you mentioned. The cost depend on the number of page (free under 5000 pages rendered every 7 days for example). For me I am at 20€ a month for my number of pages.

The best alternative is SSR, it’s free and always work, if you manage to put it in place properly no need for such services. Depending on the stack you use some tutorials are enough for you:

For me with meteor/apollo/react I didn’t have the time yet to set up a properly working SSR with fetching data but one day I will to remove prerender.

Be careful that from what I read Google is one of the “clever” crawler and sometimes managed to render properly SPA or non-ssr website, but other crawlers are generally worse than him. So if it’s “sometimes” good in google is probably most of the time bad in others…

Oh yeah, it’s horrible in the others, I was just aiming for Google for now.

I still think it’s hacky but I just remembered that I have a job now so I’ll consider prerender. Though I should note when I checked it, it was 500 free pages, not 5000. And now it seems to be 250.

As a sidenote, I also noticed that Galaxy offers pre-rendering. But when I compare their container to what I currently have, the information to compare the two is lacking and I’m not sure if it’d be a big downgrade.I’d be willing to pay a little more for Galaxy then digitalocean (Especially if it includes prerendering), but I wouldn’t want to if it’s a significant downgrade from what I have.

Galaxy:


Digital Ocean:

Slightly off topic but has anyone seen that Google Analytics isn’t properly tracking the time your users spend within your app? It seems they only track time on our landing page (the public area) but not after login.

Has anyone else notice such?

You probably need to hook the google API into your router and send pages data as they change. Do you do this?

Take at this solution, there is examples for Google Analytics and Google Tag Manger. Since this library is open source you can grab parts of the code you need, or use it as it is. We are using this solution.

Take a look on prerendering.com.
Disclaimer: yes, I’m affiliated with this service :slight_smile:
And will be happy to answer your questions.

Totally have the same issue, gives an awful bounce rate and time spent on pages. Really looking for help on this one. Will check link posted above.

For info just found this for Gtag:

Just implemented it right now, will come back with results later.

Well My question will be simple. I am using prerender.io right now, what would be the benefits of moving to this one ? If any I’m fully ready for the leap.

Would you have a tutorial or more infos about this solution ?

Thanks for confirming we’re not the only one in seeing awful bounce rate and time spend on our app.

Will check out if the solution solves the problem and improve our numbers.

1 Like

Will create a separate topic on the subject as I think we may get better inputs on the subject.

  1. It’s pay-as-you-go pricing model. On “hobby” plan for example renders will cost with no dependence to Cache Freshness (TTL):
    • 10,000 renders for $7;
    • 50,000 renders for $30;
    • First 240 renders are free.
  2. Originally made for Meteor and always tested against every release of Meteor, Angular, Vue, and React apps;
  3. With support for :nail_care: react-components;
  4. With support for :zap: AMP (accelerated mobile pages);
  5. Switch between ES5 and ES6/7 rendering engines to meet your needs with a single click in control panel;
  6. Cache-browser;
  7. “Cache upon expiration” — auto-rendering for cached pages which is about to expire;
  8. Recursively scan and render whole website;
  9. Strip or keep JS scripts and tags with a single click option;
  10. Our service focused on performance and aiming to deliver rendered results with minimal response time possible for top SEO score;
  11. Support and optimization for mobile crawlers;
  12. Using ostr.io pre-rendering service you support our open source contributions including awesome packages we maintain for Meteor.js.

Take a look on our pre-rendering panel:

2 Likes
1 Like

Hey @dr.dimitru - I’ve just signed up for ostr.io pre-rendering and successfully setup a Cloudflare Worker to handle requests. When I attempt to pre-render pages, I’m getting 200 status codes along with many 403 codes. I’m certain Cloudflare is blocking some of the requests, either from their own WAF rules or perhaps Bot Fight Mode, so I’m wondering if you have any documentation on how to whitelist the pre-rendering service?

Hey @mikeTT ,

To help you with it please send details of your setup, URL of the website, and page URLs that were successful/failed to pre-render to ostr.io support email.

Team will run tests to ensure you have proper setup.