Really hoping some Meteor/Heroku experts can give some advises.
We have a MeteorJS app, hosted on Heroku. It is already using an expensive Professional M Dyno with a lot of memory capacity. But recently, at one point of time, it triggered a lot of H12 Request time out and then Heroku presented Application error page to all visitors.
This resolved after server is restarted. But we are running out of ideas on how to prevent this issue in the future.
We kind of confirmed that there was no obvious long running tasks, like email sending, files uploading.
And then, we can no longer trace deeper what happened right before the Application Error.
To make debugging even harder, even though we can see a lot of 30 seconds response time in the charts before the application error, it means nothing to us because we have seen these 30 seconds charts everyday. We understand that this is because Meteor is using socket connection which falls back to long polling and it is natural that a lot of 30 seconds response is recorded in the chart.
This is a snapshot of the router logs of the H12 errors:
366 <158>1 2022-06-15T06:20:35.674213+00:00 heroku router - - at=error code=H12 desc="Request timeout" method=POST path="/sockjs/859/upwpunj4/xhr" host=www.website.com request_id=dea10dbf-919a-440d-afbb-87f24d6b8bea fwd="37.19.205.195,172.70.222.217" dyno=web.1 connect=0ms service=30000ms status=503 bytes=0 protocol=http
373 <158>1 2022-06-15T06:20:45.911709+00:00 heroku router - - at=error code=H15 desc="Idle connection" method=GET path="/sockjs/696/qq_433yi/websocket" host=www.website.com request_id=761c71de-0f1f-4fac-ad31-db8aed615f54 fwd="151.192.225.202,172.70.92.184" dyno=web.1 connect=0ms service=3220877ms status=503 bytes= protocol=http
398 <158>1 2022-06-15T06:20:47.934103+00:00 heroku router - - at=error code=H15 desc="Idle connection" method=GET path="/sockjs/415/mk2hh_st/websocket" host=www.website.com request_id=633aa897-ebc2-4069-b201-6aacc36dc18b fwd="2404:e801:2001:1729:dd09:de8e:a8ff:7969,172.70.142.233" dyno=web.1 connect=0ms service=1720563ms status=503 bytes= protocol=http
394 <158>1 2022-06-15T06:20:48.584366+00:00 heroku router - - at=error code=H15 desc="Idle connection" method=GET path="/sockjs/037/5fz5sxeg/websocket" host=www.website.com request_id=253e05d2-76f4-4b53-ae74-3ad252084460 fwd="2001:f40:90f:ed3:d1ea:de0:28ba:c91,162.158.163.228" dyno=web.1 connect=0ms service=1225547ms status=503 bytes= protocol=http
340 <158>1 2022-06-15T06:20:50.298552+00:00 heroku router - - at=error code=H12 desc="Request timeout" method=GET path="/" host=www.website.com request_id=59cd33a6-e155-4645-9539-8eb2e759b915 fwd="176.9.151.26,172.68.50.111" dyno=web.1 connect=0ms service=30000ms status=503 bytes=0 protocol=http
430 <158>1 2022-06-15T06:20:54.99057+00:00 heroku router - - at=error code=H12 desc="Request timeout" method=GET path="/cfs/files/images/AmZNefJYyKWBLuBdb/04.png?token=eyJhdXRoVG9rZW4iOiIifQ%3D%3D&store=thumbs" host=www.website.com request_id=8e9bb915-ea06-4698-9cf6-15f311422c82 fwd="207.46.13.66,162.158.107.186" dyno=web.1 connect=0ms service=30001ms status=503 bytes=0 protocol=http
371 <158>1 2022-06-15T06:20:55.417353+00:00 heroku router - - at=error code=H12 desc="Request timeout" method=GET path="/?pg=news/20110628_182815&id=844" host=www.website.com request_id=3d7d0829-0922-4f63-b031-a407b2d97f0d fwd="192.99.15.34,108.162.241.9" dyno=web.1 connect=0ms service=30001ms status=503 bytes=0 protocol=http
Any guidance or tips is greatly appreciated.