Hi,
I am getting the URL is not available to Google whenever I try to check the page indexing via Google search console or via Google Page Insights.
Getting this, while trying to get insights about the page
On debugging further, we get to know that whenever a page is being visited by the browser we get 200 in NGINX logs whereas whenever it’s analyzed via Google search console we get the error as
Normal Visit via Browser:
Crawl Visit / Google Page insights analysis request
It is showing a 429 error, stating too many requests, not sure why this is coming. Also, is there any standard number of requests that could be sent from the browser, is there any resolution to this issue. Please help!
In Google Search Console you have a URL Inspector. Can you see whether you have an html there or just the links to the css and js bundles as if the page is not rendered into HTML (missing SSR or prerendering)
There is an option to check the view crawled page but its disabled
Test Live URL … maybe you still get the HTML … beside any eventual error
On Testing Live URL this comes
ok do you use Prerendering or SSR?
Yes, I have configured Prerender.io also
ok, with Prerender.io you should be able to see your prerendered page. Just to make sure you actually have that page available in Prerender then you should monitor your webapp and make sure it actually redirect to Prerender.io.
In Meteor.startup you have this: WebApp.rawConnectHandlers.use(prerender). You will need to make sure crawlers end up in Prerender and not in your webapp (getting Meteor code bundles).
In prerender, its getting crawled successfully, I am able to view the crawled page, but its not working with Google search console.
Could there be any issue specific to Google Search console?
do you get your prerender page when you visit https://yourdoamain.com/?_escaped_fragment_=
No getting same 429 error code
another way to test, I am on a Mac:
- curl https://www.yourdomain.com I get the JS bundle
- curl -A “googlebot” https://www.yourdomain.com I get the proper HTML (prerendered)
Also make sure you don’t prerender https://domain.com and google crawl for https://www.domain.com unless non-www is directed to www
With 2nd command I am getting nothing, but with first I am getting the js bundle
No , rate limiter is not configured with NGINX
I wonder what generates that message. A 429 response is clearly set somewhere to be specifically returned instead of the default 502.
Do you use docker? Checking Your Current Docker Pull Rate Limits and Status | Docker
Ok, this is more realistic: Status Codes - Troubleshooting
Are you on the basic plan?! :)))
2 years back I set up my own Prerender server with Redis and never scored less than 85 in Google.
I am on AWS, currently with 2GB RAM but checked with 4GB as well, same result
Yes, Build is deployed via docker
Also,
You can consider there are two type of pages in the app, one is of profile and one is of post detail, Post detail is working fine but profile pages are having a problem
How about the Prerender package? Are you on the Basic?
Look, when you have a get from prerender by a crawler, this is what the Prerender log looks like:
Give me the HTML page so … take it.
However, when you have to prerender the page or re-prerender the page:
You have a call for each and every asset (fonts, images, libraries etc). I think this is where you encounter the rate limit.
On a post page you are under the rate limit (of Prerender probably). But on the Profile you probably exceed and this is why the page is not being prerendered (processed by Prerender.io)
