Meteor, Googlebot and SEO

Planet Earth
1

We started to see an increase in WebSocket traffic recently without a substantial increase in HTTP traffic. Here is what we figured out:

  1. Googlebot has more than doubled the crawl of javascript compared to the last week of April.

  2. Googlebot now executes Javascript in real-time. Previously, the majority of our JS crawl was through sockjs endpoints, which resulted in 404s (this was due to Googlebot crawling JS but executing them later—sometimes up to a week delay—and the sockjs endpoints no longer exist). Now, all sockjs traffic results in 200s. Blocking sockjs resulted in search console-related features of Google failing (we assume that these bots use a Googlebot that executes javascript in real-time).

  3. Google updated its documentation related to Javascript. Dynamic rendering (e.g. using Prerender.io) is no longer recommended. What is recommended: server-side rendering, static rendering, hydration.
    Google Search Makes Minor Updates To JavaScript Documentation & Drops Dynamic Rendering Workaround
    Dynamic Rendering as a workaround | Google Search Central  |  Documentation  |  Google for Developers

3 Likes
2

Interesting bit of insight. Thank you for sharing.

3

Feels like going back to the days of PHP servers with all that server rendering.

4

Still hoping for a future where client-side rendering will be enough for both good user experience and search engine indexing.

React 18 has full support for this. This requires Meteor support for renderToPipeableStream

I am looking forward to completely dropping server rendering

  • less server resources
  • less complexity

At the very least, that increase in Googlebot javascript crawl and sockjs endpoints now getting 200s instead of 404s are good signs of supporting client-side rendering

5

Thank you for sharing this information