In this talk Davide Glasser talked about one way to scale Meteor:
DDP Proxy + Application Server + Mongo Connector
IMO, this is a very good way to scale Meteor. But seems MDG didn’t follow through with this approach and used MongoDB oplog tailing approach.
Does anyone know the reason behind the decision? It would be good if MDG can still implement this architecture and optimize the mongo connector piece to avoid the shortcome of oplog tailing approach.
Yes, oplog tailing offers big performance improvement over pull and diff. But we should not settle there.
Oplog tailing is a computation expense task. When there are a lot of writes to the mongodb, all meteor processes will be busy tailing the same oplog and the application will be unresponsive. No matter how many meteor instances are added, it won’t help.
If oplog tailing task is split out and moved into a separate process (mongo connector), it will be much easier to scale the application server and mongo connector separately. The mongo connector parse the oplog once and only once, send the changes to the application processes interested on the changes. More mongo connector instances can be added in case there are heavy writes. The application processes are not affected by all these computations. They just keep processing their event loops. The application will remain responsive.
The mongo connector layer can also be very helpful for multi-tenant applications. By definition, users in one tenant will never see the data belongs to other tenants. Mongo connector layer can act as the proxy to deliver one tenant’s data to the application processes serving that tenant, not spamming all application processes like the current design.
The mongo connector layer can also be implemented in other languages better suited for computation intensive tasks, like Go or Java. This will bring additional performance boost.
So I really like the idea of split Meteor into several layers. The mongo connector layer will be very helpful for scaling Meteor.