Seeking Community Input: Meteor.js Roadmap

Some ideas from me:

  1. PWA support. Meteor already works really well for PWAs. But perhaps it would be beneficial if the guide would have a section talking about what it takes to make a Meteor app work as a PWA. Would be good to have a general service worker example.

  2. Vite. Many frameworks have moved to Vite and pretty much made it into a standard. Meteor’s Vue 3 integration uses Vite as well. Perhaps a more general move to Vite could be considered? Not really sure if that would be viable at all, but speed increases would likely be massive.

2 Likes
  1. Will be included in the Push package documentation but I agree this should be part of the guide.
    If you need a sw now, you can grab one from https://www.activitree.com/sw.js. It includes assets cacheing and is free of dependencies such as the bulky workbox.
    Android prompts for PWA installation by default. On IOS you would do something like this: A Simple React Hook to Prompt iOS Users to Install Your Wonderful PWA. | by Michael Lisboa | The Startup | Medium
    Or with some “native looks”, in React I use react-ios-pwa-prompt npm.
4 Likes

There is always a bad perception for pub/sub. New comers are using it without seeing the dark side of this beautiful feature. When the developed product is presented to the user, serious performance problems are experienced. In this case, it causes developers to move away from Meteor.js.

We both know the solution is easy. Use methods or use redis-oplog. Both of them are very easy to implement but if you are using Meteor.js for the first time. It’s always a magic / an obscurity. New developers may seem simpler to blame Meteor.js for this and switch to different framework.

Bringing Redis-oplog and Redis to the core as MongoDB could be a great.

Just to add: there is an item in the linked page about CapacitorJS, which is good (I believe a good alternative to Cordova). But the lowest hanging fruit is giving full support to PWA. We currently have a meteor app that is a web app, a PWA (desktop and mobile), an Android App (installable from Play Store), Windows App (installable from Windows Store), and an iOS app (installable from App Store) - all from the same PWA codebase.

2 Likes

Two things that are great with the implementation of Redis-oplog:

  1. You can see immediate gains just by adding the package
  2. The way to selectively support reactivity on which field of a collection makes it very efficient than supporting reactivity on everything.

Number 2 above is very powerful no matter what is the underlying technology allowing it.

1 Like

Here’s my list in rough priority order:

  1. Scaling pub / sub - Pub / sub is amazing. Imagine where it “just works” as you scale. I agree with @paulishca’s point above – replacing oplog with change streams sounds like the most promising solution based on @radekmie’s investigation Reflect on the oplog implementation · meteor/meteor · Discussion #11842 · GitHub
  2. Tree shaking - either finish the PR or move to Vite where I think you’d get tree shaking for free.
  3. Support for using Minimongo synchronously in 3.0+ - I think some solution here is important. Without it, my understanding is that the UI will need to be rewritten because it requires using the async API on the client in 3.0+. It feels unnecessary to use the async API to fetch data from Minimongo when the data is already there and theoretically can be fetched syncronously. See Support `*Local` methods in Collections API · meteor/meteor · Discussion #12644 · GitHub and zodern’s comment here.
  4. Improved Methods - I like having Minimongo as the source of truth for data on the client. Otherwise I have to devise another global store which is basically duplicating what Minimongo offers. Right now, if I fetch data via Methods, they aren’t merged into Minimongo. The idea here would be to do something akin to pub-sub-lite’s Enhanced Methods – cache Method calls (and their result data), automatically merge data into Minimongo, and emit mutation update messages from server to the caller of the Method (from). If pub / sub can scale via change streams then maybe this is not needed or at least becomes less important.
  5. Transactions - I think there were some good points made in this discussion MongoDB Transactions · meteor/meteor · Discussion #11737 · GitHub
  6. Passkeys (WebAuthn) support - I think passkeys will eventually become the primary way that users authenticate. Would be ideal if Meteor made it easy and tied it into Accounts. Accounts improvements suggestions 👥 · meteor/meteor · Discussion #11653 · GitHub
  7. Capacitor - a modern replacement for Cordova
  8. Improved offline support - Would like to see an officially supported way to cache Local Collections. It seems like we get some caching with Minimongo but it seems there is room for improvement. GroundDB was the package people used in the past but it seems to have atrophied. See Is GroundDB the best for offline storage? - #7 by brunoneumann and Improve offline support 🛌 · meteor/meteor · Discussion #11656 · GitHub
  9. Minimongo indexes and support more mongo operators in Minimongo - This also ties into using Minimongo as the source of truth for data on the client. In the past, I’ve integrated with a 3rd party API and then dumped the results into Minimongo Local Collection. It works very well but I recall having to do some hacks because you can’t create indexes with Minimongo. One solution here could be to use NeDB nedb and minimongo · meteor/meteor · Discussion #11598 · GitHub
  10. Official Jobs package - I think making wildhart’s Meteor Jobs package an officially supported Jobs package could be the way to go.
  11. Official Push Notifications package - I think making paulischa’s package an officially supported Push Notifications package could be the way to go.
  12. Bring the Meteor’s Collections API on par with the native MongoDB driver - There are some mongo methods that look helpful but currently require using rawCollection(), meaning they won’t be isomorphic.
9 Likes

As I’ve been working on Meteor pretty extensively recently, let me share what I’d like to see in the future:

  1. Investigate MongoDB Change Streams. As @jam already said, I’ve done some experiments already, but it’s unclear whether it’s always better. I expect it to be more of an opt-in thing, as it has a completely different performance characteristic.
    • That said, I still think that cultofcoders:redis-oplog should become a core package, along with oplogtoredis for seamless integration with rawCollection and external writes.
  2. Feature parity in Minimongo. That’s mostly about the methods that would truly benefit the frontend code both in terms of performance and usability – distinct and findAndModify are the best examples here.
  3. Other engines in Minimongo. Right now it’s a completely in-memory one, based on plain JavaScript objects. I’m thinking that IndexedDB or similar solutions would be truly interesting for a multitude of reasons: simplification, performance, memory usage, offline capabilities.
  4. Secondary indexes in Minimongo. That’s optional, as it’s rather unlikely to have that much data locally. Also, different engines (see above) could alleviate this problem.
  5. Better support for TypeScript. We’ve managed to bring the @types/meteor to core, but there’s still a lot to do. I think going all-in for TypeScript would be a really smart move.
    • It’d also be nice to see something like zodern:relay or grubba-rpc in core. I’m not sure how the final API should look like, though.
5 Likes

Since I’m currently maintaining it, I can get started on that, together with @radekmie and oplogtoredis. I think that as @paulishca suggested the focus should be on change streams and updating redis-oplog to become a scaling library in general as per @guncebektas suggestion.

Maybe there should be some official generator for this.

Beside the other things that have been said together with offline focus I would also like to see meteor-desktop integrated to kind of finish the set. Would be really cool also to be able to create just desktop and offline apps with optional sync to serves be it meteor-desktop or PWA installation.

I’ll think more on this and try to produce a list for the roadmap later.

1 Like

From the existing roadmap I think that all the way to next release the items are good as they are. The real discussion starts from candidate items and beyond. I have put my notes and suggestions (in no particular order) here and will update as new things come to mind.

Existing items

  • Update and fix Meteor Client Bundler or Improve DDP Client;
    Both please. First updating the bundler and fixing any issues that come up. I think a discussion needs to be had how to proceed with Vite and similar tools in this area.
    As for improving DDP client, @harry97 started experimenting with Primus, so I think there are some interesting areas that can be discussed. But in the first step it is about doing little improvements where we can.

  • Improve Passwordless package;
    No contest here.

  • Support building mobile apps using CapacitorJS;
    No contest here.

  • Bring Redis-oplog to core;
    Yes and merge with oplogtoredis. Probably the first step towards using change streams so that we can have even better scaling story.

  • MongoDB Change Streams support;
    As per above.

  • Better file upload support via DDP;
    I think we need to in general have a clear story here how file uploads should be best handled in Meteor. From basic FTP servers to AWS and specific services.

Beyond

  • Node.js v20 in Meteor v3.2 or as soon as possible and do our best to keep at the latest version.
  • Update dependencies in core. These are rarely touched, but I think they should be periodically revisited and updated to take advantage for the latest features and fixes.

Accounts

  • Accounts support for Webauthn aka Hardware keys 2FA
    Discussion
    I think this should be priority for the accounts package.

  • OAuth 2.1 support
    Need to do this sooner rather than later.
    Discussion

More relevant functionality as described on Enterprise Ready to be awesome out of the box.

  • Expand accounts functionality
    Accounts packages are probably the most popular feature of Meteor. I think we should build on that and allow additional features to help building amazing user applications.
    For example connected accounts and account switching, Apple OAuth into core, LDAP
    Discussion

MongoDB related

  • MongoDB feature parity
    We have fallen behind MongoDB functionality. This needs to be rectified as soon as possible, so that people don’t need to use rawCollection and other workarounds.

  • Minimongo improvements
    Minimongo is one of the great things of Meteor that many don’t even notice, but it needs a major upgrade when it comes to features and additional capabilties as @radekmie pointed out. I think there is a potential for collaboration with MongoDB here if we can make minimongo standalone and allow for example for data syncs between minimongo and MongoDB on server after offline period and so on.

Performance

This is kind of nebulous, but I would like to see if there is anything we can do to improve Google Lighthouse performance scores from the tree-shaking to bundler and other options. Not even close to my area of expertise so I leave this to @zodern and other experts.

Collaboration sync server over DDP

For real-time collaboration like Google Docs there needs to be extra architecture to do things like conflict resolution and other things. Yjs is one example and I think having something like that together with all the other real-time features that Meteor has would just complete the offering. Related to the next item.

Desktop, offline and PWA

I would really like to see Electron or similar integration with Meteor. Together with that also improved options on PWA and offline capabilities and caching. Maybe something along the lines of SyncedStore.

Testing

I think app and package testing need a serious look. We need better official support for things like Cypress and CI integration, not to mention stubs and guides.
Especially for local test runs it would be good to have a test DB and some nice function that would populate it with test data to be used for the tests and then it would erase it when the tests are done. Easy integration with existing test suites and tools.

Storybooks

Storybooks came out of Meteor. Right now to integrate them with Meteor is painful. At the very least we should have an extension for Storybooks to provide stubs or something to make it run. I think there is a good potential here for an extension that would allow you to set different values that could be switched around in the interface like different users and so on.

Localhost SSL

SSL is pretty much given today. So I think it is time to have it by default on localhost as well.

Browser policy packages

The browser policy packages haven’t had an update in a while to a point that other solutions started to get recommended. Personally I would like to take a look on them and give them a refresh and see if there is anything specific that Meteor can specifically offer over having a competing NPM package. If we can’t offer anything extra then we should deprecate them.

Other packages

  • logging - let’s make more use out of this package
  • callback-hook - should become the standards for hook across Meteor
5 Likes

… also

meteor

15 Likes

For me, just leave the Meteor package system and move to only npm. That’s my message in a sea of suggestions, of which I support many btw.

2 Likes

I like the Meteor packaging system and it has some unique selling points, not sure why they need to be mutually exclusive.

Voting for:

  • Passkeys
  • Improve Meteor build time;
  • Tree-shaking;
  • If it fits in fairly easily with the rest of Meteor, Vite (does this include Improving Meteor Build Time/Tree-shaking?)

Is it known yet how many Meteor apps are using SQL? If so it might be helpful to have Accounts support for SQL in addition to Mongo, especially now that mySQL/postGresQL have JSON field support.

1 Like

Just for background information to anyone interested, this is the current vite:bundler package by Akryuym that combines Meteor with Vite: meteor-vite/packages/vite-bundler at main · Akryum/meteor-vite · GitHub

As other topics have links, thought I’d link it here as well.

1 Like

I think Mongodb change stream works with oplog by itself. It’s builtin feature. Meteor also works with oplog, it’s like doing the same job.
The most advantage when using change stream api is when you scale your Meteor Apps horizontally by adding multiple Meteor instances. The oplog scanning job handled by Mongodb only, not on every Meteor instance.

1 Like

I don’t want to get into details here (we can do that in the GitHub discussion), but it’s not that obvious.

In short, Oplog generates one event for each change, whereas Change Streams one for each change for every stream. That means, Meteor would still have to go through all messages anyway (and hit the same issues as with Oplog) or have specialized streams per-publication.

The problem with the latter is that if one change impacts multiple publications (i.e., multiple users subscribe to different documents but there are some that many of them subscribe to; e.g., my friends’ profiles), then instead of processing this change once, we process it multiple times. Sure, it can be mitigated by reusing subscriptions as much as possible (I mentioned it here: 🚀 Meteor Scaling/Performance Best Practices - #3 by radekmie), but for some cases it can lead to a significantly lower performance.

2 Likes

I also vote for improving the Oplog implementation!

We are running into some performance issues with the current Oplog implementation at the moment (click here) and are looking for solutions.

We tried redis-oplog in the past but ran into some weird issues with other libraries also using redis (in our case: “bull-queue”) and had to remove it again because of that, because the bull-queue is a must have for us.
Also, since we have a “split setup” for our biggest database changes (2 separate “non meteor” plain node apps called “workers” compared to 4 meteor “main app” docker containers on a separate server) redis-oplog was also a “no go” for us because there was no really nice way to also get oplog updates “from the outside” (our workers) into redis.

We have no experience with oplogtoredis but: the more libraries and “intermediaries” you add to a “chain of things”, which all have to work in harmony and should have a good performance, the worse usually performance will become. so yeah …

At the moment we are really not sure of what is our way forward … also thinking of removing Oplog completely again …

We are using both. In our case, redis-oplog has its redis DB that is different from the bull-queue redis DB. Separate your redis DBs to prevent any collisions. Or if you are forced to use only one redis DB, use namespaces; i.e. Bull use a prefix config to minimize collisions with other apps using the same redis DB

1 Like

We’re using oplogtoredis as well as the Redis for caching ourselves and it “just works”. Actually, we use the same Redis instance for multiple Meteor applications (almost 10), and all of them have oplogtoredis set up. As @rjdavid said – it’s all about using databases (you can set it up in the connection string).

As for the additional setup – it’s a Docker container we launched some time ago (two, three years back?) and it just works.

2 Likes

These are some items I would be interested in that haven’t already been listed:

Compatibility tables / support docs
It would be nice to have someplace we can easily check:

  • which versions of the MongoDB server a release of Meteor supports
  • browser support
  • which ECMAScript version is fully supported in all arch’s. Currently, it’s not always clear if a feature we can use in modern browsers would also be supported in legacy browsers.
  • compatibility for build plugins and core packages commonly used by build plugins. Build plugins are are unique in that they are built when published and bundle a copy of their deps, so which Meteor version you publish a build plugin with affects which older Meteor versions it can run in.
  • which Node and npm version are used by each Meteor release

More thorough docs
It’s improved the last couple years, but there are still many things that are undocumented. It would be nice if someone would go through the change log at least as far back as Meteor 1.3, and the issues from the old docs repo including those that were closed for being old, and add anything missing. Another thing that would be nice is going through the docs, migration guides, and tutorials and improving the language. There are many sections with poor grammar or that explain things in a more complicated and unclear way than necessary.

Security Updates
Currently, Meteor only consistently provides security updates for the latest release. This makes it difficult for companies that are a version or two behind since they have to rush to update to the latest release before they were planning to, and sometimes run into undocumented breaking changes or bugs that make it difficult or impossible to update. I know companies that migrated away from Meteor because it was too difficult for them to stay on a Meteor release that had a node version with the latest security patches.

Also, sometimes that are delays of up to several weeks before there is a new Meteor release after there is a new node version with a fix for a security issue.

It would be nice if Meteor in some form committed to providing security updates in a way that addresses these issues.

npm installer improvements
When I created the npm installer, I tried to make it work with as wide range of node version as I reasonably could to make it easier to install Meteor. However, now it only works with a limited range of node versions, none of which are still supported by Node which makes installing Meteor more difficult. There are also a few issues open documenting long standing issues with the installer.

Windows for a while now comes with a native tar, so it would be good to use that instead of 7zip when available. This would make the installer smaller, and Windows Defender has some exceptions for the native tar which should make Meteor faster to install.

Remove ECMAScript.compileForShell
This could maybe be done by moving the function to the babel-compiler package, or moving the code to the one place it is used. This function causes every build plugin and app that uses the ecmascript package to have a full copy of the babel-compiler package, in addition to every Meteor release. A few years ago, I found that half the disk usage (I think half was ~5gb) in ~/.meteor was just copies of babel-compiler, and it was also a large reason installing Meteor was so slow on Windows.


Isobuild

Linker rewrite
This is probably one of the next things I will work on. The linker in Meteor is far behind the state of the art in terms of performance. Linkers in js bundlers have always been slow, and Meteor is missing many of the improvements other bundlers have implemented the past few years. It might even be possible to remove the disk cache, which causes its own performance issues.

Fully enable in-place builds on Windows
On Windows it is still disabled for client only builds, which for large apps can cause client only rebuilds to be significantly slower than full rebuilds. A couple years ago I finally figured out how we can safely fix this, so I or someone else just needs to spend the time to implement and test it.

Recursive file watchers on all platforms
We switched to using recursive file watchers on Windows and macOS, which is more efficient and detects file changes much quicker than how Meteor previously worked. Linux doesn’t support recursive file watchers, and the implementation used in most libraries would have created too many file watchers so it still uses the old file watching implementation. Recently @parcel/watcher added some features that would allow us to switch to it and use it on all platforms

We should also look into reducing the number of files Meteor watches. There are some folders Meteor considers as immutable, so Meteor only needs a single watcher for each of those.

Replace LRU cache
Many of the performance issues in Meteor the last few years have been from LRU caches not being configured large enough. However, constantly increasing the size of the caches will allow Meteor to use more memory than necessary for smaller apps. We should look into building a custom in-memory cache implantation that tracks which entries are needed and which can be freed so we don’t have to set an arbitrary size for each cache.

Rewrite the js-analyze code to use SWC
This part of isobuild handles scanning for imports and for globals, and over the last couple years it has become a performance issue. This would be a good place to experiment with using rust and/or wasm to speed up parts of the build process.

Switch back to using native tar
I don’t really understand the reason for Meteor switching away from it, but it seems like at some point there was a Meteor release that broke permissions in tar files created on Windows, so Meteor switched to using a js tar implementation that allows Meteor to fix the permissions when extracting the tar file on linux/macOS. Recently I found that the js tar implementation is significantly slower, and makes downloading a Meteor release or Meteor packages up to 2 - 3 times slower. It would be worth reviewing this and finding a way to switch back to using native tar.

noticed that the last few years, there hasn’t been as much public information on the reason for changes compared to before. The detailed PR’s and commit messages that Meteor had for many years is what allowed me to understand how and why Meteor worked the way it did and to start contributing to it.

Regarding switching to vite, I’m not sure that is the best path forward. Each bundler makes very different trade offs, and vite and isobuild are very different. A recent example is top level await. Having async code in the root of modules was very common with fibers, so we spent a lot of effort into creating a spec-compliant implementation for Meteor. In comparison, vite uses a different implementation in development and production, and the production implementation behaves differently and is not spec compliant.

It seems that the two main things missing from Meteor, besides features that we can add, are:

  • performance
  • vite has a larger ecosystem, at least in part to having better support for build plugins

For performance, I have a long list of things we can improve before we even need to think about a new architecture. Its been a few years since we’ve had a release focused on build performance.

For build plugins, it is surprisingly difficult to create a build plugin for Meteor - there are many bugs and it is difficult to create a well working plugin, and there is very little documentation. Besides that, Meteor’s current design for build plugins is from Meteor 1.2, and I’m not confident that being designed around batching has aged well. It probably is time to design a new, simplified build plugin api.

9 Likes