In our Meteor 2.15 project, some ‘server only’ methods do quite a lot of collection queries and updates, resulting in very slow server responses, up to tens of seconds. Therefore, I recently added stubs to achieve ‘optimistic UI’ and speed up the client UI experience. This resulted in a drastic improvement of speed: tens of seconds on the server turned into a few hundreds of milliseconds on the client… wow, what a difference!
There is only one big problem now: we found that collection insert _id’s for the same documents don’t always match between simulation (client) and server.
When both client stub and server create exactly the same (amount) of collection inserts, they do match, however in some of our methods the client simulation may intentionally skip certain inserts compared to the server, resulting in unmatched _id’s for the same inserted documents. Even worse: the _id of inserted document A on the client may be equal to _id of inserted document B on the server in these cases.
Is there a way to solve this?
If we can’t guarantee identical _id’s for the same inserted documents on both client and server, we can’t use ‘optimistic UI’, which would be a real pity now we’ve seen the huge performance benefit.
I haven’t tested this scenario specifically but I think the above options might still get the job done. Let us know how it goes for you. Curious, why do you intentionally skip certain inserts?
My methods are async, so now I use Method.callAsync() on the client, in conjunction with zodern:fix-async-stubs to solve the need for an await on each Method.callAsync(). I don’t think I can use Meteor.apply or jam:method with async methods on the client, right?
The client has a limited set of documents on certain collections, it does not have all documents loaded, due to e.g. user permissions, that’s why lesser (manipulated copies of) documents may be inserted compared to the server.
You mentioned that your inserts can last “tens of seconds” in the server. Zodern’s solution also mentioned some limitations. You might be hitting those limitations.
Option “{ returnStubValue: true }” doesn’t seem to work either: I would have expected 2 console.logs now, however still only got 1.
I tried with removing the ‘zodern:fix-async-stubs’ package, but it didn’t make a difference either.
Coincidence or not: the latest Typescript Meteor.d.ts file does not include Meteor.applyAsync() definitions (yet?), while Meteor.callAsync() is defined. Could it be that the ‘returnStubValue’ option is not supported anymore, or a bug is causing this unexpected behavior?
In a nutshell: our application allows users to record and playback various types of media on a scheduled timeline. We call these media events ‘parts’, which actually contain metadata to how the media should be played, e.g. when, how, and for how long. Media can be audio, video, images, animated text etc. The media itself is not stored in the database, only the metadata in the ‘parts’, which also include references to the media files. These ‘parts’ are organized into ‘tracks’, which on their turn make up an entire ‘arrangement’.
The user can edit these parts (and tracks) on a visual timeline, e.g. selectively move, copy, delete or modify a group of parts on the timeline.
As a ‘copy edit’ example, we measured the time needed by 1 server-only method to copy 328 parts. This translated into 656 document reads (findOne by document _id) and 328 document writes (insert), so nearly 1000 database operations. On the server this took about 14 seconds. By comparison, the client stub (with same code on client and server) took 134msec to do the same number of operations… more than 100x faster… wow.
However, depending on e.g. user permissions and/or preferences, some parts or tracks in the currently loaded arrangement may not be available at the client. Still some ‘global timeline’ based edits may be launched then, e.g. “vertical cut and copy all parts across all tracks at a specified time range to a new time range”. This is an example of a situation whereby the client - using the same method - executes a subset of database inserts compared to the server. It is in these situations that we noticed that the generated insert _id’s go out of sync between client and server.
So I conclude that Meteor can only generate identical id’s if the client and server execute the same amount of inserts within a method. I haven’t found any documentation on this, so I don’t know whether this is normal expected behavior or a bug.
Meteor uses a fairly simple system for the id’s to match on the client and server. When calling the method on the client, Meteor creates a seed for the random id generator. It uses the same seed when running the method on the server. This means that the first id created in the method on the client and server match, the second id matches, etc.
Three possible solutions are:
if possible, you could refactor the server code so it first creates the documents that would be created on the client, and then insert the documents that are only inserted on the server
if it is possible for the client to estimate how many documents the server will create and when, the client could create an unused doc (the collection doesn’t matter) instead or use the api Meteor uses to generate an id, with the goal of matching the order the docs are created on the client with the order they are created on the server
if what docs are created is based on the client state, you could maybe send some of that information when calling the method
Though from your last post, it seems the first two suggestions might not be possible.
I wanted to clarify some things regarding zodern:fix-async-stubs. As far as I know, it doesn’t have any additional limitations than what already exist with async stubs. Rather, the type of async stubs that it warns against writing are even more problematic when not using zodern:fix-async-stubs.
One reason they are more clearly marked as not supported when using zodern:fix-async-stubs is if you don’t have them, then all of the problems with async stubs disappears. I was also comparing them to what a perfect solution would look like, and trying to figure out how it could be clearly documented if it was integrated into Meteor.
This doesn’t sound right to me but if it is right, from the UI perspective I would not expect optimistic UI via reactivity but via local DB with a push/pull data mechanism in the background. I am thinking of that experience in Google Docs when you update a document and you can see a message at the top such as “your document has been saved”. I think you can also think of it like an offline-first kind of application. Get everything you need at the beginning (read) and only push your updates back to the server. When you write to the server you only expect success (boolean), not data. If no success, you are in the “something went wrong and we could not save your data” kind of experience and your UI shows your last updates although the user understands that is not the last saved version. On Cancel, you get the last saved version on the server.
Ok, what I mean to say … from what I understand of your process, Meteor reactivity is no the solution.
A typical large arrangement can easily contain 1000 parts (e.g. 20 tracks with each containing 50 parts). Each part corresponds at least to 1 document, so edits of this kind of magnitude are not exceptional. I don’t see how this could be handled differently, as the user can select randomly a group of parts to be edited.
Your suggestion about an ‘offline first’ application is indeed tempting, having thought about that too, but I don’t immediately see this feasible in our application. E.g. we also use FFmpeg, GraphicsMagic and other heavyweight media processing libraries, they then would need to be run at the client either. And the application is meant to be used on smartphones too, so we try to keep the client as light as possible in terms of memory, e.g. audio is recorded at the client in linear pcm, and converted at the server into compressed audio for playback of multiple tracks in parallel. We would soon run out of memory on a smartphone if we’d keep all audio as linear pcm on the client too during ‘offline’ mode. I am not saying it is impossible, but it would require a large refactoring of the current code.
Ok, what I mean to say … from what I understand of your process, Meteor reactivity is no the solution
You are probably right, however this would mean forgetting pub/sub and fetching everything via methods, right? But then I’d loose the Minimongo functionality and differential updates from the server I suppose? I am not sure how to implement an non-reactive scheme without having to rewrite a complete alternative for the pub/sub mechanism.
The Redis route is also something I had in mind as a possible alternative, however I don’t know how much performance gain we would get. Certainly not a factor 100x as we achieve now with the optimistic UI approach I assume? I was considering using ‘cultofcoders:redis-oplog’ in combination with radekmie’s ‘changestream-to-redis’ and ‘protectAgainstRaceConditions: false’ to achieve best performance, but this is all very new, and so is my knowledge on all this… I am bit hesitating to dive into it (yet).
“I was considering using ‘cultofcoders:redis-oplog’ in combination with radekmie’s ‘changestream-to-redis’ and ‘protectAgainstRaceConditions: false’ to achieve best performance, but this is all very new,” - I think they are pretty much drop in libraries, not a lot to change in the project except adding a Redis server on a local IP in your VPN.
What I meant with “offline-first” was only considering data. You would still do all your processes on the servers but you would rely more on methods (as you mentioned) and a local global data store on the client. In other words, you replaces pub/subs + minimongo with Methods + local store (such as Redux - perfect for both web and React Native).
Ok I guess what I am trying to say is that you will always need a source of truth and that is the server. If the server operations are slow, the truth is … late and that is a problem. You will need a lot of connections to do all those operations (Mongo is not really like 1 user - 1 connection) and a lot of power to stay subscribed to this amount of data with multiple users.
Data integrity is also a problem with this amount of writes. I wonder if you use Transactions.
It looks like you want a “write behind” mechanism (like in the Redis video above) but Minimongo is your in-memory DB instead of Redis.
I would start by focusing on having those DB operations optimized and work on data integrity and server load strategies and add that Redis Oplog.
I was not aware of transactions to be honest, I’m new to using databases actually.
When wrapping a sequence of database read/writes in one transaction, apart from the data integrity aspect, does that also mean that other client’s concurrent server methods still see the ‘old’ documents until the transaction is done?
Would that also mean that the reactive pub/sub mechanism only updates the client side Minimongo in an ‘atomic’ way after the transaction ended?
Currently I put mutexes around my methods so that no other method can interfere with a current sequence of database updates for a given arrangement. But wrapping the whole method 's sequence of updates in one transaction would yield much more integrity indeed. And is also ‘atomic’ if I understand well so I could didge my mutexes then too?
Looks like ‘jam:mongo-transactions’ is an easy way to add transactions. Thanks for pointing me in this direction!