Latency compensation is a really nice feature of Meteor that truly improves the UX of your apps, but we have entirely removed it from our code not too long ago, as it was hard to maintain client and server shared code, without everything feeling messed up. We’re starting to feel the slow-down from a pure server-side approach though, and we’d love to have some of that sweet compensation-juice improve our user experience.
I’m following up on this very old thread: Best practices for pub/sub, latency compensation and methods in real-life scenarios? - #3 by waldgeist which includes @waldgeist 
We face some of the same issues raised there:
- Difficulty of maintaining a shared code base between client and server, without spreading isServer/isSimulation
- Bundle size management, as all this code will take up space
- How could you code-split this properly, to avoid loading all of your meteor methods code on initial load?
- Hiding of server-side business logic for certain methods
- How do you organize your codebase properly such that you have server-only, and client+server code, without rewriting a lot of things everywhere? Again, on the server you don’t care about code-splitting, but on the client, it’s a must-have!
A few pointers:
- Perfect separation of security and business logic
- Maintain a very small set of frequently used CRUD methods that exist on both client and server, and keep the rest on the server
- Create a shared Class of business logic code on client+server, that you subclass on the server to share functionality
I’d love to see a really elegant wrapper around meteor methods that handles this in a scaleable way.
We’re currently using this cool wrapper by @diaconutheodor: GitHub - cult-of-coders/mutations: A new way to think about mutations inside Meteor, but we should enhance it to allow it to deal with this issue in a better way.
2 Likes
I don’t have a complete answer for you, but couple of notes, perhaps out of order.
- Load data from methods instead of pub/sub - you’ll lose “free” latency compensation.
- Use
mdg:validated-methods for better separation of security and method logic.
- Only link your methods to the server bundle.
- Client side - use a separate data store, not connected to the stores used on the server. I like to use
Ground:DB directly so that the data will persist. Drive your views out of this collection.
- Fill client data stores manually using methods.
- For optimistic updates, use a Flux like action pattern.
More on that last point. Create your client side CRUD actions (create, read, update, delete). These will:
- Update the local store (
Ground:DB)
- Capture old value, update client side value in collection, then attempt to update the remote collection over a method.
- If successful, sync data to grab the new _id, and other autofields.
- If not successful, rollback #2, and throw an error to be handled however.
- Other sync as necessary.
This is essentially the algorithm I use with npdev:collections in PixStori, except without trying to hide the validation source (I figure if the security is right, it won’t matter if the hacker can see the code). You should be able to create some tools to help you reuse some of the logic in your CRUD methods.
2 Likes
Agreed on the Flux/Redux approach to optimistic UI. I use Vuex w/Vue, most data is retrieved via validated-method and store in Vuex. All UI pulls data from the Vuex store.
When calling an action to fire off a request to the server I immediately update the data in the store to pretend the server request worked (whether it’s updating a document or deleting something, etc). Once the request to server is done I update the store if it didn’t go successfully. It’s more boilerplate code but you get full control over the experience.
Well, why would you want to replace the built-in latency compensation and rewrite your own? You have to add a ton of code to make sure your server-side mongo updates have the same effect as your client-side store updates.
If your codebase is already pretty large, adding in a whole client-side store, with replicated CRUD sounds like a massive investment, whereas I’m hoping to be able to get some meteor latency-compensation with a few small refactors.
Well, why would you want to replace the built-in latency compensation and rewrite your own? You have to add a ton of code to make sure your server-side mongo updates have the same effect as your client-side store updates.
A long list of reasons. These are just a few
- Optimistic UI requires a subscription and data stored in Minimongo. Meteor pub/sub is very resource intensive and can cause serious scaling issues if a lot of your local data is from subscriptions. I avoid pub/sub whenever possible as most of the time it’s really not needed.
- Having MongoDB queries in your client code is such a tight coupling with a single database, it’s one of the biggest design failures of Meteor IMO. I prefer to abstract this away and make all database requests on the server in a ValidatedMethod or REST route, so that if/when I need to start moving some data outside of Mongo or to another microservice, I can do that without entirely rewriting the client. Decoupling has serious benefits down the line.
- With more boilerplate comes more control. You gain much more granularity over how you want to handle optimistic updates. Some actions can assume they failed after 1 second, some can assume they failed after 10s. You can decide whether to optimistically update the UI immediately, or just show a loading indicator for some pieces of data that are more appropriate to show a loading state.
TL;DR Meteor pub/sub and optimistic UI is great for rapid prototyping, but as you being to scale and build enterprise solutions with Meteor it’s often beneficial to remove some of the Meteor magic and gain more control over your application’s behavior, and avoiding a very tight coupling with Mongo is a good idea for future-proofing.
2 Likes
I wonder if you can get the simulated methods to do this for you, if you include the methods in your client bundle. The way to do it would be to do what npdev:collections does - it has a factory method that creates on the server a Mongo.Collection and on the client creates a Ground.Collection instance. The factory method in npdev:collection is only used to allow a single import point, that gets you a different instance on server and client. Once you have that collection, import it in your method definition files, and then it should update your ground db collection (or whatever you provide, if you make your local package), client side only, and roll back if the server result doesn’t match.
All this assumes I have the optimistic workings of methods correct in my understanding.
Or you could just do it manually - in your client side method call, use an action pattern - update the local collection, run the method, then either roll back if it fails, or leave it if the method succeeds. It should be relatively easy to create some sort of tooling to make this easy to do for multiple collections’ CRUD methods.
I’m still a big fan of pub/sub and optimistic UI, but then again I don’t have that many users yet and most of my data does benefit from being reactive.
I don’t worry too much about polluting shared client/server methods with Meteor.isServer, because A) security by obscurity is a fallacy (provided any private api keys are securely stored in Meteor.settings.private, etc); and B) I dynamically import only the methods I need for a particular route or component, so size isn’t really an issue.
For the dyamic import I store all my shared methods in /imports/methods/ grouped into logical files for dynamic importing. e.g. I have all my admin methods (which are only used by logged-in admins) in one file, and then import that file in the main admin component. The server imports all of the files.
For this reason, my landing page which consists of only links to other routes does not contain any methods at all in the initial js bundle.
I do also have some server only methods which only make sense on the server, e.g. credit card payments, etc. These are all in /server/methods/... and so automatically imported into the server bundle. I prefix the names of all these methods with server. so it’s obvious that they are server only.
If you really wanted to hide your business logic, or minimize the size of the js bundle or dynamic imports, then you could embed submethods within your shared methods:
Meteor.methods({
'doSomething': function() {
// validate auth and parameters...
// then...
Meteor.isServer && Meteor.call('server.doSomething.pre', ...);
// do something public to update the local UI quickly
// then...
Meteor.isServer && Meteor.call('server.doSomething.post', ...);
return something;
},
Alternatively, there’s nothing stopping you from importing completely different method code but with the same name on the client compared to the server - as long as they do the same thing from the client’s POV.
One way of doing it is by keeping your collections flat and simple. This could be done by making it more relational and ref stuff by id.
Very interesting takes @wildhart, thanks!
We’ll probably go with adding a few select methods to the client, especially the update methods, so that we can easily get some latency-compensation without a lot of effort.
We happen to not have the constraint of scaling to tons of users, as our apps provide a lot of value to a very small amount of users. So CPUs are always at low usage