Hi there, I’ve been debugging a server method that validates and cleans a potentially large set of data from the client, in memory, and then asynchronously inserts it into the db using
Collection.rawCollection().insertMany(). It works pretty well, but on large sets of data (10000+ rows), it can take up to a minute and the client will lose connection to the server, usually resulting in the method being called over and over again in a loop of:
- The client calling the method for the first time and losing the connection
- The client reconnecting to the server and calling the method again (and by this, I mean the connection gets re-established and without the user doing anything, the method is called again, as if Meteor thinks the method call didn’t go through and retries by default)
- And losing connection and so on and so forth…
Obviously this is a huge problem and is preventing me from releasing this feature, because it will overwhelm the collection with duplicate data as soon as a user tries this in production.
Side note: I’m still using the older pre-2.0
aldeed:simple-schema to validate and clean my stuff.
I guess, in this situation, I have two questions.
Is this behavior normal for Meteor? I’ve tested lengthy server-side executions in the past and I’ve never seen this problem of the connection timing out and the method call getting stuck in a loop before.
Can anyone please suggest a better, faster way of doing this? The db call doesn’t take long, but iterating through the dataset and validating/cleaning is taking far too long than it should.