Client-server connection timing out during long Meteor.method executions

help
1

Hi there, I’ve been debugging a server method that validates and cleans a potentially large set of data from the client, in memory, and then asynchronously inserts it into the db using Collection.rawCollection().insertMany(). It works pretty well, but on large sets of data (10000+ rows), it can take up to a minute and the client will lose connection to the server, usually resulting in the method being called over and over again in a loop of:

  • The client calling the method for the first time and losing the connection
  • The client reconnecting to the server and calling the method again (and by this, I mean the connection gets re-established and without the user doing anything, the method is called again, as if Meteor thinks the method call didn’t go through and retries by default)
  • And losing connection and so on and so forth…

Obviously this is a huge problem and is preventing me from releasing this feature, because it will overwhelm the collection with duplicate data as soon as a user tries this in production.
Side note: I’m still using the older pre-2.0 aldeed:simple-schema to validate and clean my stuff.

I guess, in this situation, I have two questions.

  1. Is this behavior normal for Meteor? I’ve tested lengthy server-side executions in the past and I’ve never seen this problem of the connection timing out and the method call getting stuck in a loop before.

  2. Can anyone please suggest a better, faster way of doing this? The db call doesn’t take long, but iterating through the dataset and validating/cleaning is taking far too long than it should.

2

I guess http://docs.meteor.com/api/methods.html#DDPCommon-MethodInvocation-unblock will make you happy.

3

this unblock() will not help here. It doesn’t make the method run or return any quicker. All it does is allow the same client to make another request to the same method on a new Fiber.

That allows you to “parallelise” that method to some extent. However, misuse of this.unblock() can cause really hard to debug errors if you’re not aware of how it works.

I’m currently in the middle of writing an article on common misconceptions of methods - and this.unblock() features in that!

1 Like
4

This has been documented in the forums before. I can’t recall right now whether any definitive solution for it came out.

One option is to use a separate (micro)service to do this - so not part of your current client/server code - a sort of service worker approach.

Is it worth examining the validation/cleansing to see if that can be optimised?

1 Like
5

If anything, this behavior goes against what the default should be for a method execution. It’s behaving as if I have called this.unblock() and re-running whenever the client re-establishes the connection.

6

Thanks for the responses, @robfallows. You’re definitely right, besides finding a general workaround for the timing out/method call looping issue, the only way I can comfortably get past this is by speeding up the validation function. Another temporary solution would be moving validation to the client to bypass the timeout issue, but that does nothing to make it more efficient, and potentially exposes the database to exploitation.

I’ve got some work ahead of me, it looks like: further researching and diving into the method timeout problem, and figuring out some ways to either offload parts of the method to another service or just straight up go back to Programming 101 and see if I can come up with a faster way to validate/clean the data.

1 Like
7

I’ve managed to get the validation/cleaning execution time down to ~20s on almost 20000 documents, which is pretty alright. I’m still looking for ways to do better but at least it’s semi-reasonable, at least for an administrative feature. You can follow that story here…

However, I’m still having the aforementioned DDP timeout and method call looping problem, and when I switched to long-polling by disabling websockets on my http-proxy config, I actually got a stack trace this time in the client (without long-polling it was failing silently, and only by calling Meteor.status() would I have known)

So I’m just going to put this here while I contemplate opening an issue…

WebSocket connection to 'wss://localhost/sockjs/339/omftwo1n/websocket' failed: Connection closed before receiving a handshake response
WrappedWebSocket @ VM43:35
SockJS.websocket @ ddp-client.js?hash=bc32a16…:1363
SockJS._try_next_protocol @ ddp-client.js?hash=bc32a16…:1285
SockJS._didClose @ ddp-client.js?hash=bc32a16…:1193
SockJS.that._ir.onfinish @ ddp-client.js?hash=bc32a16…:1119
EventEmitter.emit @ ddp-client.js?hash=bc32a16…:239
xo.onfinish @ ddp-client.js?hash=bc32a16…:2108
EventEmitter.emit @ ddp-client.js?hash=bc32a16…:239
that.xhr.onreadystatechange @ ddp-client.js?hash=bc32a16…:936

Have any thoughts @robfallows or anyone else?

8

Hi guys, i actually am experimenting the same issue and it’s quite an issue how to increase the timeout.It should be possible to modify it just for the sake of the current project one is working on … a solution?

9

Basically on the offending method call, you can call Meteor.apply with the noRetry option, so that the client doesn’t re-run the method.

What I ended up doing, however, is refactoring that whole part of my app so the client does not need to participate in the data entry module I was setting up. It might be worth thinking about reworking whatever method or functionality you’re trying to achieve that’s causing you the issues. I think using noRetry should be a last resort because it often results in bad user experience either way.

1 Like