Help Tiny: Post Your Most-Wanted Features for Meteor

vlasky · October 10, 2019, 12:02am

Most wanted features for Meteor:

Feature 0: Release Meteor 1.9 with Node.js 12 support ASAP

A much faster V8 JavaScript engine and support for worker_threads, Bigints and TLS 1.3 provides performance, scalability and security benefits to everyone.

Please allow @benjamn to keep working on this without interruption as he is very close to completion.

Feature 1: Improving MongoDB Oplog Tailing Mode Scalability with Minimum Result Fetch Interval Setting

This is critical for improving the scalability of Meteor apps that use MongoDB in Oplog tailing mode

The Meteor MongoDB code needs a throttling feature to set a minimum time interval between successive result set fetches for a given reactive query. This allows the developer to imposes a hard limit on the maximum update rate of a given reactive query.

This would greatly reduce CPU and memory usage for reactive queries and reduce network bandwidth when those reactive queries are published and subscribed to by clients.

In many cases, it is not necessary to fetch updates at top speed in response to every change. For example, if we are using a Mongo publication to update a user interface component like a table or map or chart on a web browser page, we gain nothing from updating it more than say once per second.

A new option minResultFetchIntervalMs would be added to Mongo.Collection.find(), which represents the minimum allowable time delay in milliseconds between successive result set fetches for a given reactive query.

For example, a publication that can send reactive updates at a maximum rate of once per second would have a minResultFetchIntervalMs of 1000. A maximum rate of twice per second would be a minResultFetchIntervalMs of 500 and a maximum rate of once every 5 seconds would be a minResultFetchIntervalMs of 5000 and so on.

Equivalent functionality has existed in the mysql-live-select package, the key component of the Meteor MySQL integration since the beginning. It has been crucial in allowing our reactive Meteor MySQL apps to scale. I feel compassion towards people using MongoDB and also want them to have a similarly good experience like we currently have with MySQL.

I have written this up on Github:

github.com/meteor/meteor-feature-requests

Improving MongoDB Oplog Tailing Mode Scalability with minResultFetchIntervalMs

opened 06:50AM - 08 Oct 19 UTC

vlasky

Project:Mongo Driver

This is my proposal for an enhancement to the Meteor MongoDB code to improve the… scalability of Meteor apps that use MongoDB in [Oplog tailing mode](https://blog.mlab.com/2014/07/tutorial-scaling-meteor-with-mongodb-oplog-tailing/) This would further add to the excellent work done by @benjamin and @theodorDiaconu The Meteor MongoDB code needs a throttling feature to set a minimum time interval between successive result set fetches for a given reactive query. This allows the developer to impose a hard limit on the maximum update rate of a given reactive query. This would improve scalability by greatly reducing CPU usage and memory used for reactive queries and network bandwidth when those reactive queries are published and subscribed to by clients. In many cases, it is not necessary to fetch updates at top speed in response to every change. For example, if we are using Mongo publication to update a user interface component like a table or map or chart on a web browser page, we gain nothing from updating it more than say once per second. A new option **minResultFetchIntervalMs** would be added to Mongo.Collection.find(), which represents the minimum allowable time delay in milliseconds between successive result set fetches for a given reactive query. For example, a publication that can send reactive updates at a maximum rate of once per second would have a minResultFetchIntervalMs of 1000. A maximum rate of twice per second would be a minResultFetchIntervalMs of 500 and a maximum rate of once every 5 seconds would be a minResultFetchIntervalMs of 5000 and so on. Equivalent functionality has existed in the [mysql-live-select package](https://github.com/vlasky/mysql-live-select#additional-settings), the key component of the [Meteor MySQL integration](https://github.com/vlasky/meteor-mysql) since the beginning. It has been crucial in enabling our reactive Meteor MySQL apps to scale. The lack of this feature in the Meteor MongoDB code is the biggest remaining obstacle to making reactive Meteor MongoDB apps scale. It should be incorporated without delay. Example: Let's imagine our Meteor application displays a map with real-time vehicle locations which are stored in a MongoDB collection published by the server. ``` Meteor.publish('vehicleLocations', function() { return Locations.find(); }); ``` Let's imagine that this collection receives 100 separate vehicle position updates in one second. That would result in 100 extra entries inserted into the MongoDB oplog. In the current Meteor MongoDB code, that would result in the publication potentially being triggered by each oplog entry, sending up to 100 updates to each subscribed client, resulting in lots of network bandwidth, CPU time and memory being needlessly consumed. **How this would be improved with minInteval:** Instead, let's imagine that we could publish the collection and specify a minResultFetchIntervalMs of 1000ms (1 second): ``` Meteor.publish('vehicleLocations', function() { return Locations.find({},{minResultFetchIntervalMs: 1000}); }); ``` At time=0, oplog entry 1 causes the result set to be fetched, but then no further result fetch is allowed to take place until 1000ms (1 second) has elapsed. Between time=0 and time=1, Meteor's MongoDB observer code notices oplog entries 2-100, but will not take any immediate action. Instead, it will schedule the next result set fetch to occur at t=1. The same scenario repeats itself for the remaining 9 seconds of activity. At the end of the 1 second, only 2 result set fetches would have been performed instead of 100. **Answers to Expected Questions:** Q. So you are ignoring events in the oplog. How is that good? A: They are not being ignored - we just don't react to each one of them - kind of like when someone rings your doorbell multiple times - the first ring is enough to set you in motion towards the door. <br/> Q: How is this more efficient than just using poll and diff? A: This approach avoids needless polling and provides predictable response times to events. <br/> Q: What is the scalability limit with this approach? A: How quickly the Meteor MongoDB observer code can read the oplog. For best performance, one would store their MongoDB database & oplog on an SSD (preferably NVMe).

Feature 2: A Meteor accounts package that is decoupled from MongoDB.

Please introduce an abstraction layer in the code to decouple the database code from the Meteor account logic.

One simple starting option is to modify Meteor accounts to use a virtual/overridable Collection class instead of Mongo.Collection, so the developer can use their own database-specific code to implement collection methods like find(), insert(), upsert(), update(), remove() etc.

Feature 3: Minimongo secondary index support

At the moment, there is a third-party partial implementation of Minimongo secondary indexes provided by the experimental package helfer:minimongo-index. It only speeds up point queries but not sorting operations, but perhaps this code could be used as a starting point for a complete implementation.

This has been on the feature requests list for a long time:

Feature 4: Renew MDG commitment to Blaze

Back in August 2018, @hwillson brought a lot of comfort to the Meteor community with the following statement on behalf of MDG. I urge Tiny to renew this commitment.

Feature 5: HTTP/2 Websocket Support

Node.js V10.12 added RFC 8441 extended connect protocol support to allow use of WebSockets over HTTP/2.

The main performance benefit is that a single HTTP/2 connection can multiplex requests and WebSockets.