First of all, I applaud your effort here and agree with many of your points. I’ve been thinking about your core question quite a bit as well. I think there are a couple other points to consider in tackling the scalability issue:
- Meteor must get rid of oplog tailing.
- I think managing subscription state should be reconsidered now that the landscape has shifted quite dramatically from when Meteor was first created. More thoughts on that here: Rethinking Meteor – managing subscription state
I think your proposed changes can be done completely under-the-hood so there are no changes to the current Meteor public APIs or at least that should be the goal imo.
Regarding your specific points:
100% agree. There was a GitHub discussion some time ago on making the move to something else.
Interesting. I don’t know much about CBOR but benchmarks would be interesting to evaluate.
Meh. I think Meteor should use JSDoc for all the same reasons that Svelte migrated from TS to JSDoc. Functions exported from Meteor core can and should still have all the same benefits of TS that you’re used to (typechecking, intellisense, inline documentation etc).
My position is, types are fantastic, TypeScript is a bit of a pain … as soon as you use a .ts file, then you have to have the tooling to support that … there’s all of these points of friction when you use a non-standard language like TypeScript that I have come to believe makes it not worth it. So instead, we have put all our types in JSDoc annotations, and we get all of the type safety, but none of the drawbacks, because it is just JavaScript, everything is in comments, you can just run the code. This is what we do in the Sveltekit codebase and it has worked out fantastically so for Svelte 4.0, we’re going to do the same for Svelte because it’s going to enable us to move much more quickly.
– Rich Harris
Let application developers use TS if they’d like. My other belief is that in time TS will phase out much like Coffeescript. I expect that JS will eventually adopt the types as comments proposal. We’ll see though.
I think this is one possible solution and maybe a really good one. I think it’s also possibly worth exploring something similar to ElectricSQL – essentially a service that sits between your Meteor app and MongoDB and uses change streams. It could even be used as a caching layer so that MongoDB is hit relatively infrequently and be written in another language that is built for scalability and concurrency, e.g. Elixir. Perhaps this is similar to radekmie’s DDP Router but I’m not really sure since details were limited. My hypothesis is that jam:pub-sub could get most apps relatively far where they wouldn’t need to reach for either Redis or an Electric SQL equivalent, both of which bring in more complexity and I believe would only truly be needed at large scale.