Seeking Community Input: Meteor.js Roadmap

announce
50

Definitely, agree. There seems to be a confusion on what redis-oplog brings to the table. Yes, it can replace and replicate the features of meteors oplog-tailing or change stream. But the real strength of redis-oplog is the granularity it can provide in tracking changes which brings a huge performance gain. It can track a single key on a single document if needed (at the most extreme as an example) without tracking the entire collection for changes.

Unless change stream can do such with the corresponding performance gains without tracking all changes in the entire collection, then redis-oplog deserves a core implementation.

Change stream might reach that point. But if ever it would, it’s not there, yet.

51

There are 2 critical issues for our team:

I. Native support for Vite or equivalent technology.

Reasoning:

  1. Performance. Developing apps with rich UI has become nearly impossible because native HMR/refresh is way too slow. We had to wait 5-10 seconds for the changes to be visible in the browser. We use 2 workarounds:
    – Develop front-end separately using pure Vite. It provides great DX, everything is lightning fast, but that’s not how we want to use Meteor. We still have to make time consuming merge of the front-end and back-end.
    – Use a fork of Akryum’s vite:bundler. We have to use a fork because the package itself has unresolved critical issues related to Meteor packages. DX is ok but it’s still worse than with pure Vite: HMR is slower, page refresh is often required due to HMR errors, usage of Meteor packages is limited.
    None of the solutions are good.
  2. Ecosystem. Vite allows us to use any front-end framework without additional effort.
  3. Tree-shaking.

With that in mind, we consider Vite’s native support to be Meteor’s most important missing feature. We are ready to contribute.

II. Overall build/rebuild/refresh performance.
As projects grow, we still experience a lot of performance issues on Windows, even after all the possible optimizations. Using Vite partially solves the problem but without Vite it’s a nightmare. Rebuilds are too long, memory usage is high, running Meteor slows down VS Code a lot. The initial build also takes a lot of time for a big project. We would also like to have faster production builds. Again, with Vite it’s not that bad, but modern JS frameworks are far ahead in this regard.

6 Likes
52

I do agree with that, having vite will enable meteor to just works with standard front end tools.

1 Like
53

Any change that Angular (Typescript based) will be officially supported by Meteor?

This was possible even before Meteor switched to typescript using a 3rd party compiler from Urigo, but its unmaintained for 2 years now.

54

Hey everyone,

I just wanted to say thank you for sharing your thoughts and ideas with us on the Future of Meteor.JS. Your input is incredibly valuable to us as we work on defining our Roadmap. We’ll make sure to carefully consider all of your suggestions. Thanks again for participating!

9 Likes
55

Something that I have not seen mentioned is adding MongoDB aggregation to Meteor’s Mongo.Collection class.

Using rawCollection is sufficient, but I think we could get better APM insights?

5 Likes
56

The reason we have the wrapper around Mongodb is Minimongo. An implementation of aggregation pipelines in Minimongo does not make sense. With aggregation pipelines we can take the data from a collection and completely transform its structure. When we sync that transformed Data with Minimongo we can’t run that transformed Data through the same pipeline again (unlike with the usual Mongo queries).

2 Likes
57

Doesn’t look like anyone mentioned this, so I’ll add on!

I think that Meteor’s modules and ecmascript system is fairly out of date now, and quite a source of complexity that keeps Meteor in legacy, especially as time moves on. It was built during a time when a revolutionary set of language features were introduced but before engines supported the new features. It was necessary in that time.

Today, the language churn has slowed down compared to the ES2015 transformation of JavaScript. The majority of the revolutionary and transformational language features used today are supported in all engines, especially the features that modules+ecmascript were originally designed to support, ES Modules being one particularly huge feature.

I believe the best way forward for Meteor would be to entirely scrap the current module system, and start anew with a system designed specifically for today’s ES Module reality.

The current system takes ES Modules, and compiles them into a Meteor’s custom CommonJS-like module format, going against the grains we have today.

This was necessary back in early 2016, when no engines supported ES Modules. But if we were starting anew today, we’d rather build on top of ES Modules natively, allow people to write native ES Modules, and optionally allow a bundling step with tools like Rollup or ESBuild, and optionally allow enabling down-leveling (transpiling) newer language features to lower targets, both options exclusive of each other instead of coupled together like currently (where currently we handle language transpilation on top of the module system).

Right now the system (by default) compiles modern syntax into an ancient format. This is not very necessary today because most language features today are small increments compared to ES2015, and most code today can be comfortably written without transformation for all modern browsers. And IE is done for (:partying_face:), so we’re free from some shackles we previously had.

This isn’t to say we don’t need transpilation, but in many cases it is not needed. Having a simple and clean setup that would allow people to choose their targets in a way in which they can avoid as much transpilation as possible, without the complexity of the current module system, would be great. This new system would be opt-in, instead of opt-out like current system. Modules and bundling would be a totally separate option from language transpilation, both optional and serving only for optimization and support of older browsers.

To make this happen, we may want to take advantage (as an opt-in) of newer tools like Vite, or Astro on top of Vite, in order to provide an ES-Module-oriented system in which we also get hot module reloading and bundling (Vite uses hot-reloading ES Modules in dev mode, and Rollup for production bundling).

In fact! Maybe the baseline should just be vanilla ES Modules out of the box (with isobuild still around for back compat (i wish it weren’t)), and any tools built on top of that would need to handle vanilla ES Module code intelligently. This is the future that is currently being explored by tools, and it would be nice to be on this path starting with vanilla ES Modules.

Whatever steps we take, all the optimizations should be opt-in. The reason for this is because if we can write in vanilla ES Modules and our apps work, and then do we choose to apply optimizations, we can rest assured that our code will work for a long time without having to upgrade build systems, etc, in the future. When things work with the new vanilla standards, they will be supported for a very very long time (JS engines are not likely to remove ES Modules any time soon, browser engines not likely to remove established ways of loading assets, etc). Vanilla ESM web apps will keep running forever with very little dependency maintenance, if at all.

By starting from a fresh ES Module foundation, we’ll eliminate a bunch of complexity that I assume no one here actually wants to maintain, and we’ll then be set up for longevity and compatibility with the wider JS and web ecosystems.

The only problem is the work needed to take a step back. There may need to be sacrifices made in backwards compatibility in order to be aligned with the future of JavaScript. Various packages, including vite:bundler , edemaine:solid, svelte:compiler, barbatus:typescript, etc, rely on the modules package, but modules needs to be gone for Meteor be following a JavaScript-standards-aligned future path, so this change will not be a simple upgrade.

However, without this change, Meteor is going to be stuck in legacy.

3 Likes
58

I think one way we can help move forward is fork Meteor, strip out all the legacy, and:

  • build from the start on vanilla ES Modules
  • possibly replace Blaze as the default with a simple and standards-forward Custom Element solution (I’ve made an alternative to Lit which serves as an example for making such a thing, not very much code)
  • or perhaps use (or make something like) Astro that has great SSR out of the box (including client-side routing) for most popular frameworks (and custom elements!)
  • what to do with the package system? The only thing it is currently really useful for is build plugins, but if we go vanilla ESM, we don’t need to start with a build.
  • If we want to support JSX (React, Solid, etc) we need a build.
  • Maybe we keep Meteor’s package system for back compat to load packages that expose globals (and may rely on a build), but by default a new app will start with no build stuff
  • We probably need builds, but the default should be simple, without one.
  • if anything, a build system should understand a vanilla ES Module website to and be able to bundle it, for production, similar in concept to Parcel (but Parcel is old and has its issues, needs rethinking from scratch and focused on vanilla ESM from the start)
  • Maybe we write an HTTP/3 (QUIC) ES Module server that can push modules to the client (HTTP/2 removed HTTP push, but HTTP/3 brought it back in a better way)
  • Maybe we make a separate NPM registry specifically for Meteor packages, as a way to not only use the industry standard NPM, but to also have a separate showcase of Meteor specific packages?
  • while we’re at it, make the TypeScript support more ideal (currently some hoops to jump through to get all things well typed for both packages and meteor code itself)

Etc. There are lots of things to consider.

Maybe an incompatible fork is the only way to fully experiment to see what would be the ideal new setup, where we strip out the stuff we know we don’t want (f.e. the current ecmascript ESM emulation that was so great in the day, but is now hindering us) and go from there.

Such a fork may even help visualize what actually needs to be done in Meteor core if it isn’t mergeable back in. Or maybe it becomes its own thing. Either way it would help bring the JS community forward.

Anyone interested to experiment with this? With the right set of people (namely people that can dedicate time to making it happen), we can settle on what exactly to do, then do it. I might be able to find a few hours average per week or two.

4 Likes
59

Sounds like we can make a fork like this for Deno. I think that could be more worthwhile. Right now I don’t have the time nor interest to go this deep. But after we get few versions into Meteor 3 I would be interested in getting in on looking on the next major versions.

I like the idea of our own NPM registry to keep the Meteor packages separate. Best if we could figure out some migration for existing packages, I think that would require migration of Meteor capabilities to NPM packages, so that you can do:

import { Accounts } from '@meteorjs/accounts-common'

or something like that.

4 Likes
60

This is an excellent point that I didn’t really appreciate until recently.

I think Bun would also be an interesting experiment at some point.

A Meteor SDK is also intriguing. Then I suppose you could bring any framework - Astro, SvelteKit, etc. But I’m not sure if there would be a big tradeoff in “it just works so I can focus on business logic” which I think is one of Meteor’s best selling points.

2 Likes
61
  • Testing with modern testing tools (Jest, Playwright, React Testing libraries, etc.)
  • Move off of Atmosphere (or open source it)
3 Likes
62

Something popped up in my mind few days ago, but if Meteor Software is still intent on bringing redis-oplog to the core, @diaconutheodor expressed intent in helping out at this endeavor so try reaching out again and see if is still interested. There’s no one better to work on this integration since the guy who built it!

4 Likes
63

Backwards compatibility or migration for existing packages does sound nice, but it’s an aspect that could be time consuming and there are a number of packages presumably relying on ecmascript by now too, so it will be difficult.

I think at least a fork not focused on back compat (at least initially) would allow a group of people to spend time on the important future-forward ideas. Then this fork can help inform what Meteor actually needs to do while keeping back compat, or perhaps the back compat can be bolted on after the new concepts are in place.

Of course not having a community be able to move forward could be a bad thing. At the same time, if the new thing is really good, it’ll find the community regardless of back compat or not (f.e. just like new things like Astro have).

This will be a free time effort for me if I do find time (unless I can get hired, I’m available for part time at the moment) so with that in mind I would personally want to do this exploration without back compat taking away time from implementing the concepts.

On another note, I think Meteor needs its own Showcase to really really sell itself:

1 Like
64

Yeaaaaaah, really can’t wait to get rid of Fibers. I’m now getting this error in an app with innocent normal async/await code:

AssertionError [ERR_ASSERTION] [ERR_ASSERTION]: Cannot await without a Fiber

I didn’t want to troubleshoot, so I reverted back to Meteor.bindEnvironment (but this means I’ve made work for myself when the Meteor update that removes Fibers lands).

65

That looks like the correct error :sweat_smile:

1 Like
66

That’s a Meteor error. Vanilla Node.js doesn’t have this issue.

I simply tried to use a Promise (returned from Meteor) like you would in plain Node.js, but it fails.

This just gets in the way of writing modern code like with any non-Meteor Node.js app.

67

Yep. We are all awaiting here with the Promise of Meteor 3.0

68

Hi, I remember someone published a similar topic in the past, so my list was this one Question for Meteor Developers - #27 by diavrank95 . However, some of the items have already been resolved in the last few months, so, I’ll leave here only the missing ones and I will add a few ones:

  • Upload files as form-data over DDP
  • Transactions support for Accounts methods like createUser, sendEnrollmentEmail, sendVerificationEmail, etc. Mentioned here.
  • Official support for code coverage in meteor test (not sure if there is already an official package for code coverage)

Things that I’d change about Meteor

  • One thing that I’d change is how to subscribe to publications since currently, you need three things in the client side to invoke a publication: publication name, parameters, and the collection which is used to return the cursor of the publication. So, I’d like that publications only receive the publication name and its parameters, without specifying the collection. So, I imagine invoking publications in this new way:
//Client side
const subscription = Meteor.subscribe('users.list', params);
const reactiveData = subscription.findData(selector, options);//or just use: subscription.reactive() if no longer possible to use parameters with this new change.

I don’t know if this change is possible, but this would fix an issue that maybe many beginners have had when they are learning to use publications, which is the combination of data when 2 publications use the same collection and they’re invoked by the same client at the same time.

Features for Meteor 3.0+

I have been working in my current job with Nest JS and I can say this is a very good framework for building web apps (actually APIs) based on REST. It is very similar in some way to Spring Boot and Laravel frameworks because have a well-defined directory structure and entities which allow to developers follow the same pattern to build apps. So, in Meteor we have in the documentation an example directory layout but it could be much better if it copies things from other frameworks like Nest. I know that Meteor is an isomorphic framework, so, I guess that is the reason why it can have different directory structures but maybe it’s better to have a standard structure that allows new Meteor devs to have a concrete shape of Meteor framework to build apps and at the same time in an even faster way.

So, this is a list of features that Meteor can copy from other frameworks:

  • Have a standard directory structure
    • I use Meteor with Vue in the same project because Meteor allows to have both directory structures, for the back-end and front-end in the same app. So, the standard directory structure which I mean, it’s especially for the backend which it should include folders to store files like middlewares, controllers, publications, services, dtos, models, helpers, constants, etc.
  • Create some commands for meteor-tool to create file templates (for back-end only). For instance: meteor make:controller products and a file is created with the validated method. So far, I have in mind commands to create controllers (validated methods), publications, services (business logic), and models (data layer). Also, it will be needed a flag to create a meteor app with the standard layout. For example: meteor create myapp --standard
  • Use Typescript as the programming language for standard meteor apps.
    • Nowadays, it’s well-known that many companies are using TypeScript instead of pure JavaScript, since you can avoid bugs by using data types and this helps to build scalable big applications.
  • Include Astronomy for standard meteor apps.
    • Nowadays, all web frameworks use an ORM (Object Relational Mapping) for the data layer. Just to mention a few examples: NestJS uses TypeORM, Laravel uses Eloquent and Spring Boot uses Spring Data (based on Hibernate). Although Meteor uses a NoSQL database with MongoDB (which has primary support), its ODM (Object Document Mapping) is Astronomy since AFAIK is the only one created for Meteor and it works very well. The only thing it needs is to add transactions support in the main repository, but I have done a fork and I published a new version of this package in Atmosphere , however, it will need its typings of TS with this new version since the original typings only work with original meteor package which is jagi:astronomy. So, in short, it would be nice to add support for an ODM of Meteor, either Astronomy or a new one created by Meteor Software.
  • Use NPM as Meteor’s only package manager.
    • When I started to learn Meteor, I wondered what was the reason that Meteor didn’t use NPM as its package manager and I found that it was because Meteor was released when NPM was in its first versions, so, maybe MDG wanted to create its own package manager because NPM was not as developed at that time. Nowadays, npmjs is the platform with the largest number of packages (more than maven, composer, pip, among others), so, I think if Meteor uses NPM as its only package manager, more benefits will come, for example: more nodejs/javascript developers could contribute to Meteor by creating only npm packages because it would no longer be working on a single Meteor ecosystem, but rather it would be using possibly packages that use other Node frameworks (such as Nest, Express, Electron, React Native, etc.). Also, another benefit it would be that deployments of meteor apps would be faster with an implementation of other NPM modules which are used by other frameworks like Express or Nest. Maybe this is a difficult change because it would mean migrating all meteor packages to NPM, but if we start to standardize meteor applications, we can identify the packages that would be worth migrating to NPM.

Lastly, I think the API for Meteor Methods can be improved by using OOP paradigm (by using classes and inheritance at each layer. e.g. controllers, services, models, dtos, etc) and decorators from TS.

@Controller('users') // in this part we could specify the prefix for all the method names defined in this controller. e.g. users.create, users.update, users.delete OR /users/create, /users/update 
export class UsersController extends Method {

   //...maybe it will be needed a class constructor to inject the services that we want to use in the controller.  

   @Name('users.create')// or like a route '/users/save'
   @Validate(CreateUserRequestDto) // Request DTO is a class that has the structure of the endpoint's parameter. In this kind of DTOs we can take advantage of the `Match` package to validate the parameters.
   @ResponseDto(UserCreatedResponseDto) // Response DTO is a class that has the structure of the endpoint's response
   @BeforeHooks(functionX, functionY)
   @AfterHooks(functionZ)
   createUser(user: CreateUserRequestDto){
       return this.usersService.createUser(user); // Returns a user object which will be used in the Response DTO to take the properties that are needed to return to the client.
   }

  //more meteor methods ...

}

The above approach would be a proposal to implement an N-Layer Architecture for standard apps in Meteor.

69

That might be possible for a high-level API. Not sure if this is possible in general because publications can return one cursor, multiple cursors, or a handle. See Publish and subscribe | Meteor API Docs.

I must say I would only give this as a recommendation, and maybe part of a scaffold tool. But I definitely do not want this structure, libraries or tools to be imposed on me.

Yes.

Yes!

I’ve always been limited by what Meteor’s API had to offer in terms of publications and methods. A lot of people use additional packages to fill the gap, some use ad-hoc libraries. I would say that it would be nice to have a native API which includes validation etc. but it would perhaps be more interesting to first have low-level public APIs to allow us to roll our own more easily when we need it.

2 Likes