Help Tiny: Post Your Most-Wanted Features for Meteor

mitar · October 9, 2019, 10:49pm

Yes, even internally they are very close so. Even so much that one can implement Tracker in terms of Vue.

vlasky · October 10, 2019, 12:02am

Most wanted features for Meteor:

Feature 0: Release Meteor 1.9 with Node.js 12 support ASAP

A much faster V8 JavaScript engine and support for worker_threads, Bigints and TLS 1.3 provides performance, scalability and security benefits to everyone.

Please allow @benjamn to keep working on this without interruption as he is very close to completion.

Feature 1: Improving MongoDB Oplog Tailing Mode Scalability with Minimum Result Fetch Interval Setting

This is critical for improving the scalability of Meteor apps that use MongoDB in Oplog tailing mode

The Meteor MongoDB code needs a throttling feature to set a minimum time interval between successive result set fetches for a given reactive query. This allows the developer to imposes a hard limit on the maximum update rate of a given reactive query.

This would greatly reduce CPU and memory usage for reactive queries and reduce network bandwidth when those reactive queries are published and subscribed to by clients.

In many cases, it is not necessary to fetch updates at top speed in response to every change. For example, if we are using a Mongo publication to update a user interface component like a table or map or chart on a web browser page, we gain nothing from updating it more than say once per second.

A new option minResultFetchIntervalMs would be added to Mongo.Collection.find(), which represents the minimum allowable time delay in milliseconds between successive result set fetches for a given reactive query.

For example, a publication that can send reactive updates at a maximum rate of once per second would have a minResultFetchIntervalMs of 1000. A maximum rate of twice per second would be a minResultFetchIntervalMs of 500 and a maximum rate of once every 5 seconds would be a minResultFetchIntervalMs of 5000 and so on.

Equivalent functionality has existed in the mysql-live-select package, the key component of the Meteor MySQL integration since the beginning. It has been crucial in allowing our reactive Meteor MySQL apps to scale. I feel compassion towards people using MongoDB and also want them to have a similarly good experience like we currently have with MySQL.

I have written this up on Github:

github.com/meteor/meteor-feature-requests

Improving MongoDB Oplog Tailing Mode Scalability with minResultFetchIntervalMs

opened 06:50AM - 08 Oct 19 UTC

vlasky

Project:Mongo Driver

This is my proposal for an enhancement to the Meteor MongoDB code to improve the… scalability of Meteor apps that use MongoDB in [Oplog tailing mode](https://blog.mlab.com/2014/07/tutorial-scaling-meteor-with-mongodb-oplog-tailing/) This would further add to the excellent work done by @benjamin and @theodorDiaconu The Meteor MongoDB code needs a throttling feature to set a minimum time interval between successive result set fetches for a given reactive query. This allows the developer to impose a hard limit on the maximum update rate of a given reactive query. This would improve scalability by greatly reducing CPU usage and memory used for reactive queries and network bandwidth when those reactive queries are published and subscribed to by clients. In many cases, it is not necessary to fetch updates at top speed in response to every change. For example, if we are using Mongo publication to update a user interface component like a table or map or chart on a web browser page, we gain nothing from updating it more than say once per second. A new option **minResultFetchIntervalMs** would be added to Mongo.Collection.find(), which represents the minimum allowable time delay in milliseconds between successive result set fetches for a given reactive query. For example, a publication that can send reactive updates at a maximum rate of once per second would have a minResultFetchIntervalMs of 1000. A maximum rate of twice per second would be a minResultFetchIntervalMs of 500 and a maximum rate of once every 5 seconds would be a minResultFetchIntervalMs of 5000 and so on. Equivalent functionality has existed in the [mysql-live-select package](https://github.com/vlasky/mysql-live-select#additional-settings), the key component of the [Meteor MySQL integration](https://github.com/vlasky/meteor-mysql) since the beginning. It has been crucial in enabling our reactive Meteor MySQL apps to scale. The lack of this feature in the Meteor MongoDB code is the biggest remaining obstacle to making reactive Meteor MongoDB apps scale. It should be incorporated without delay. Example: Let's imagine our Meteor application displays a map with real-time vehicle locations which are stored in a MongoDB collection published by the server. ``` Meteor.publish('vehicleLocations', function() { return Locations.find(); }); ``` Let's imagine that this collection receives 100 separate vehicle position updates in one second. That would result in 100 extra entries inserted into the MongoDB oplog. In the current Meteor MongoDB code, that would result in the publication potentially being triggered by each oplog entry, sending up to 100 updates to each subscribed client, resulting in lots of network bandwidth, CPU time and memory being needlessly consumed. **How this would be improved with minInteval:** Instead, let's imagine that we could publish the collection and specify a minResultFetchIntervalMs of 1000ms (1 second): ``` Meteor.publish('vehicleLocations', function() { return Locations.find({},{minResultFetchIntervalMs: 1000}); }); ``` At time=0, oplog entry 1 causes the result set to be fetched, but then no further result fetch is allowed to take place until 1000ms (1 second) has elapsed. Between time=0 and time=1, Meteor's MongoDB observer code notices oplog entries 2-100, but will not take any immediate action. Instead, it will schedule the next result set fetch to occur at t=1. The same scenario repeats itself for the remaining 9 seconds of activity. At the end of the 1 second, only 2 result set fetches would have been performed instead of 100. **Answers to Expected Questions:** Q. So you are ignoring events in the oplog. How is that good? A: They are not being ignored - we just don't react to each one of them - kind of like when someone rings your doorbell multiple times - the first ring is enough to set you in motion towards the door. <br/> Q: How is this more efficient than just using poll and diff? A: This approach avoids needless polling and provides predictable response times to events. <br/> Q: What is the scalability limit with this approach? A: How quickly the Meteor MongoDB observer code can read the oplog. For best performance, one would store their MongoDB database & oplog on an SSD (preferably NVMe).

Feature 2: A Meteor accounts package that is decoupled from MongoDB.

Please introduce an abstraction layer in the code to decouple the database code from the Meteor account logic.

One simple starting option is to modify Meteor accounts to use a virtual/overridable Collection class instead of Mongo.Collection, so the developer can use their own database-specific code to implement collection methods like find(), insert(), upsert(), update(), remove() etc.

Feature 3: Minimongo secondary index support

At the moment, there is a third-party partial implementation of Minimongo secondary indexes provided by the experimental package helfer:minimongo-index. It only speeds up point queries but not sorting operations, but perhaps this code could be used as a starting point for a complete implementation.

This has been on the feature requests list for a long time:

Feature 4: Renew MDG commitment to Blaze

Back in August 2018, @hwillson brought a lot of comfort to the Meteor community with the following statement on behalf of MDG. I urge Tiny to renew this commitment.

Feature 5: HTTP/2 Websocket Support

Node.js V10.12 added RFC 8441 extended connect protocol support to allow use of WebSockets over HTTP/2.

The main performance benefit is that a single HTTP/2 connection can multiplex requests and WebSockets.

korg0001 · October 10, 2019, 12:09am

NativeScript integration. It would be nice to be able to connect to a Meteor server over DDP using NativeScript.
Support for the latest version of Mongo and aggregations without meteorhacks:aggregate.
I don’t think there’s a need for shiny new tutorials, but rather maintaining the ones that exist. I pulled the latest todos and npm audit shows 53 packages with vulnerabilities and tests are failing. It would be nice if there was an updated version of this classic tutorial for Blaze, React, Vue, iOS, and Android. I would also like to second an updated version of Discover Meteor.
Official Discord meteor community. I appreciate and have greatly benefited from Vue’s Discord communities. It’s easy to ask for help, assist others and stay up to date on changes. The Meteor Chef is in maintenance mode and had a popular Slack channel that is mostly inactive. Ownership of an unofficial Discord community was handed off last month and it’s pretty quiet in there too. If one was official, maybe it would be more active.

harry97 · October 10, 2019, 5:48am

Spot on! There are lots of conflicting opinions on how to proceed and which changes/features to prioritize. So, let the pros decide.

marklynch · October 10, 2019, 8:28am

Thanks, that’s good to know. We’ve checked them out a few times but the thing that’s always put us off is the data centre they use isn’t part of one of the big-3, so we couldn’t launch our atlas DBs there, so it means we’d have to contract server & DB with them. I agree with you though about supporting the smaller guys, if only it was easier. BTW - I love that they have a public roadmap on trello

alawi · October 10, 2019, 8:37am

Here is another potential strategy:

Reduce FUD by giving community clear commitment for long-term core support
Update the site/brand to explicitly endorse React, Vue and Svelte (and Blaze for backward compatibility)
Fix the pricing of Galaxy to appeal to entrepreneurs (at least match DO price), and decouple hosting from monitoring for extra revenue
Empower the open source contributors and package maintainers to handle the other features (I suggested adding feature voting page with sponsor influencer points)
Partner with companies that uses Meteor for contracting (e.g. Cult Of Coders) and support major open source Meteor projects (Vulcan,Wekan, RocketChat etc.) they’re your marketing front
Grow & Profit

thatgibbyguy · October 10, 2019, 1:03pm

That was actually a thread I made, not sure Evan ever really endorsed it considering he left MDG just a month or so after the post. However, yes, meteor should have official Vue.js support.

captainn · October 10, 2019, 5:00pm

I actually looked in to the internals of Tracker to see if something could be adjusted there to make programming for React’s concurrent mode a bit easier (right now it’s super hard for a couple of reasons). I found an inversion of control that makes that difficult - each “Dep” maintains its own internal list of observers at the dep itself, not in the computation. I thought (without going deep enough to know if it’ll really solve the problems I’m trying to solve) that if the deps were either moved to the computation directly, or maybe if we replace the dep store with a weakSet or something like that, we might be able to rope the JS GC engine in to do cleanups for React’s discarded renders. The idea being that if the only references to those observers (or to the computation) is in the discarded react instances themselves, they should eventually get cleaned up by the GC. But that would all take a lot of testing - and either a fork of Tracker, or at least an extension.

rjdavid · October 11, 2019, 12:45am

The reverse was also possible. Evan endorsed it heavily internally, MDG made its position clear (similar to what he posted), and then he left because of conflict of interests

opscaptain · October 11, 2019, 1:23am

OpsCaptain already provides Meteor.js hosting via Git or CLI deployment and it is only $7 for a 1GB container and you also get a 1 GB MongoDB only $2 more expensive than DO. Infact, we are the cheapest Meteor.js hosting on the market. If Galaxy pricing is an issue, that shouldn’t be a problem for entrepreneurs as OpsCaptain is available. Just saying. Sure we do not have a fancy metrics graph and do not scale out but we are very reliable and get the job done for entrepreneurs just starting out. We are working on the scaling out and load balancing part.
https://www.opscaptain.com/pricing

alawi · October 11, 2019, 4:55am

That is good to hear, but the point is to bring money back to Meteor so it can further be developed…

arggh · October 11, 2019, 8:52am

Npm packages are fully supported, but Atmosphere packages are a lot more powerful and provide actual value. I would not give up Meteor’s own package system.

Blaze works and is designed for Meteor’s reactivity system. Lots of Meteor apps in production are using Blaze. Unless Blaze somehow becomes a major blocker in adopting some core improvements to Meteor itself, I don’t see the point in dropping official support for it, other than making lots of paying Galaxy customers very upset.

I could always appreciate faster rebuilds, but to me they are pretty darn fast already. I’d like to hear more about your wish for zero config, since Meteor pretty much is… zero config? Or is it not? HMR would be great, even if it’s implementation would have some deficiencies.

Agree! I still cannot understand why they didn’t use this synergy to their own benefit.

My own wishlist

Tree shaking / dead code elimination
Meteor-quality developer friendly support for web workers and service workers, with mitigations against the most common footguns
Stick to the commitment towards backwards compatibility like we’re used to: which means also not dropping support for Blaze
Modernizing core packages, mongo first comes to mind
Better marketing, community engagement, social media interactions and just… communicate.

In general, I think a nicely paved path towards smaller, leaner bundles would be a great move. This would open up Meteor as a viable platform for more than just “web apps”, the domain where Next, Nuxt, Gatsby and soon Sapper are the current go-to’s.

Last but not least, why Galaxy doesn’t provide a managed database that just works with Meteor is… beyond me

jabid · October 11, 2019, 9:23am

Hey guys, who wants to volunteer to summarize all posts? With the frequency of the same requests like Vue+Meteor (42), React (19) Blaze (2) …

illustreets · October 11, 2019, 10:07am

Perhaps that would be a mistake. There have been many opinions which mention no actual package/library - do we leave those out?

Narrative and the collective voice of community expressed under this thread are more important than a (dare say) misleading list of packages.

illustreets · October 11, 2019, 10:16am

I’m sure you are aware of the work Cult of Coders have done regarding the integration of Apollo with Meteor. I agree, it’s not an official package by MDG, but it works very well and it is supported by one of the best Meteor developers out there: https://github.com/cult-of-coders/apollo

harry97 · October 11, 2019, 10:56am

Well said, @arggh

jabid · October 11, 2019, 11:49am

okay. I agree … opinions are the most important. Also, make syntheses, it can be effective for the next steps …

storyteller · October 11, 2019, 12:29pm

142 posts… I guess this is a good procrastination as any. Give me some time guys.

mhinton · October 11, 2019, 1:29pm

Has Tiny posted anything anywhere about what they plan to do with Meteor? The fact we haven’t seen anything along the lines of “Hey we are Tiny, we love Meteor and want to push it forward.” is making the company I work for consider switching to Nuxtjs/Apollo. Subscriptions would be more work but then would only be used where absolutely necessary.

opscaptain · October 11, 2019, 1:34pm

That still can be done with OpsCaptain around. It will be impossible and misguided for whomever runs Meteor to think they can control all the monetization avenues. For example Wordpress has their hosting and thriving but it doesn’t mean there are not like some 5 other dirt cheap Wordpress hosting companies out there. Believe it or not a $7 USD per app hosting strategy if MDG had pursued would have been a waste of their time because you are talking about a min of 10k apps to even make something decent. Which is simply a number of apps that does not exist in the Meteor.js community. MDG knows this but you guys sitting on the forums talking don’t. MDG is very smart do not underestimate this simple fact.

So yeah MDG can focus on the Apps generating revenue, OpsCaptain will focus on the apps just getting started and require a tight budget. I think this in the end helps the Meteor.js community than harm it.