Illegal instruction (core dumped) node main.js

one of my prod meteor servers core dumped today showing

2021-03-19 17:20:48.714 UTC  Client-side invalid-state error at / (with no user):
		InvalidStateError: Failed to execute 'transaction' on 'IDBDatabase': The database connection is closing.
		transaction@[native code]
		https://medssenger.net/1d2e7e2e1a0cc0ce4eecba377b3b110e002c4fe8.js?meteor_js_resource=true:27:3422
		https://medssenger.net/1d2e7e2e1a0cc0ce4eecba377b3b110e002c4fe8.js?meteor_js_resource=true:1:8267
		promiseReactionJob@[native code]
		in build 2103191331
		on Mozilla/5.0 (iPhone; CPU iPhone OS 14_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Mobile/15E148 Safari/604.1


#
# Fatal error in , line 0
# Check failed: IsFeedbackVector().
#
#
#
#FailureMessage Object: 0x7f7790bc56a0
 1: 0xa5c711  [node]
 2: 0x19b4ac5 V8_Fatal(char const*, ...) [node]
 3: 0x1a098a6 v8::internal::compiler::JSFunctionRef::feedback_vector() const [node]
 4: 0x1a39cce void v8::internal::compiler::PipelineImpl::Run<v8::internal::compiler::GraphBuilderPhase>() [node]
 5: 0x1a421bc v8::internal::compiler::PipelineImpl::CreateGraph() [node]
 6: 0x1a42d9e v8::internal::compiler::PipelineCompilationJob::PrepareJobImpl(v8::internal::Isolate*) [node]
 7: 0xc347dd v8::internal::OptimizedCompilationJob::PrepareJob(v8::internal::Isolate*) [node]
 8: 0xc39248  [node]
 9: 0xc39b60 v8::internal::Compiler::GetOptimizedCodeForOSR(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::BailoutId, v8::internal::JavaScriptFrame*) [node]
10: 0x100b8f5 v8::internal::Runtime_CompileForOnStackReplacement(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x13a71b9  [node]
/tmp/boot_nodejs.sh: line 84:    16 Illegal instruction     (core dumped) node main.js

file /tmp/boot_nodejs.sh is my launcher wrapper where I launch node server

node --version
v12.16.1

meteor --version
Opening db file /inner_home/egtved/.meteor/package-metadata/v2.0.1/packages.data.db
Local package version is up-to-date: meteor-tool@1.10.2
Opening db file /inner_home/egtved/.meteor/package-metadata/v2.0.1/packages.data.db
Meteor 1.10.2

its a new error … this is an actively maintained setup which I have had in prod for several years doing periodic code releases infact just did a prod push a few days ago … all my dev servers have been running clean … in above error it says :

InvalidStateError: Failed to execute 'transaction' on 'IDBDatabase'

however my mongo has no such db so can only guess its an internal meteor or nodejs or mongo system db ???

am tempted to post over on GitHub - nodejs/node: Node.js JavaScript runtime unless someone has a better suggestion

cheers

Oops, that looks like an IndexedDB error. IDBD is a browser database. In the case of Meteor, it is used for caching dynamic imports.

It rings a bell. I saw this before, here: "Failed to execute 'transaction' on 'IDBDatabase': The database connection is closing." – what?. The OP describes how he traced the origin of the error. It may be worth asking @peterfkruger how he debugged it.

As noted by @illustreets, I’ve run into something similar indeed a few times. My app’s Meteor version is still 1.9.2. I ultimately did nothing to fix this, because apart from a few times this error never popped up again.

The exact same errors in my app never resulted in server crashes, however, so your case seems far more serious than was mine.

Let’s separate concerns at this point. The server crash is one issue, posting it on Github sounds like a very good idea.

The other concern is the IDBDatabase transaction error. Please read my original post and the answers to it if you didn’t yet.

I’ve checked the error stack trace of your app, and this revealed that the error occurred in your app at the very same Meteor code line as in mine:

In the original thread @rjdavid said they’re experiencing this error too.

The error occurs in conjunction with Meteor’s dynamic import handling, so far usually (or only?) observed on iPads and iPhones. It breaks the dynamic import cache in IndexedDB for that app, meaning that the app on that device can’t recover again, unless the IndexedDB will be properly deleted and the app restarted. I’ve found out about how to do that programmatically:

window.indexedDB.deleteDatabase("MeteorDynamicImportCache");
window.location.reload();

See that answer here. In that scenario your client code would catch the error and deal with it accordingly.

I still can’t comprehend how a mere propagation of a client error to the server can lead to its crash. Are you 100% sure that the two things are related to each other? Does the timestamp of the server crash match very closely that of the client error?

2 Likes

We still receive this issue from our client logs twice or thrice a week. We are at Meteor 2

Our solution is to display an error to our user (Sorry, this component failed to load) and a 5s counter which automatically executes the deletion of the idb and then reload the page (precisely as the solution posted by @peterfkruger). We no longer have a case wherein the client never recovers but the client error still happen

But this is strictly a client issue for us. iOS issue to be precise

Cool forensics! :smiley:

Regarding the error popping on the server, it could be that the OP is running something server side which can trigger that behaviour. Maybe a headless browser. Perhaps an iOS emulator or such.

2 Likes

How are you catching it? Using the catch branch of the dynamic import promise? We have not encountered it but Like to cover it in our next deployment as precaution.

Assuming you use something like react-loadable, one possible approach is to leverage Loadable's loading property, which is a component that receives an optional error property (among others).

In your own version of Loading component you can evaluate the property error, and if it is an InvalidStateError (with a message “Failed to execute ‘transaction’ on ‘IDBDatabase’”) you can take action to recover a presumably broken dynamic import cache in IndexedDB.

Beyond informing the user that the app needs be restarted due to an error one could provide them with a button [RESTART APP], or do what @rjdavid did, render a countdown and take action automatically.

The action itself can be as easy as noted above, just those two statements.

Other than that, if you use import(), you can surely use a catch block or function and do the evaluation there. This raises the question of how you inform the user about what happened. Also if the error persists, it can cause an endless loop.

We are indeed using the error props to handle this issue

Does that mean it is not recommendable to use dynamic imports with iOS for now?

I can say it happens less than 1% of the time to less than 1% of our ios users.

@rjdavid – “error props”? How are you trapping that error? We report client-side errors to the server using the global onerror handler. We could scan the text of the error in there and reset the IDB accordingly. Are you (effectively) doing that or able to trap it more narrowly?

We are using the loading component of this package

@rjdavid – thanks for the clarification :slight_smile: