Node-fiber NOTE OF OBSOLESCENCE

[macrozone]

@alawi

I hope you are not proposing that, can you clarify this statement?

meteor is kindof stuck in DDP+pub/sub+Meteor.Collections. kadira voted for a new datalayer end left the community because meteor could not move forward. mdg itself adapted graphql with apollo but could not make it the default for meteor. The community itself got splitted. Some want to keep the things the way there are and that is ok. But the world keeps spinning and so does the ecosystem. Deprecation of fibers is another nail in the coffin for the classic meteor datalayer.

So either:

• meteor adopts or creates a new datalayer and deprecates the classic stack. This has been tried without success
• meteor finds a way to transition the classic datalayer to a async/await compatible variation. This would require to start finding and deprecating code patterns that won’t work without fibers. This could be achieved by forcing promises in some situation. This could be done start to warn now when this is not done. Similarly how react restricts certain patttern in save-mode.

When thinking about it, its probably doable as it only affects server code. The problem is, that much code is shared between server and client.

[alawi]

@macrozone

meteor is kindof stuck in DDP+pub/sub+Meteor.Collections

Well, I don’t see it that way, you can use GraphQl today with in Meteor environment, and a lot of folks migrated to this data layer already.

But I personally I like the simplicity of websockets, DDP and meteor methods. Not every app will scale to be facebook and require their API flexibility and scale, most apps will run just fine with web sockets/DDP.

This classical Meteor deprecation and changing it completely has been debated to death on this forum. I personally like it the way it is today plus some improvements. And frankly I believe any major refactoring and breaking change won’t happen, it is just not feasible at this stage, it takes tons of investment and there is tons of code/products/businesses built on top of the existing system already. Like I said, we’ve to be pragmatic and opinionated and not try to morph Meteor to something that is not.

I think we should keep the discussion focused on what to do with Fibers in the short-term.

[macrozone]

@alawi

I think that there is a good chance Meteor 2.0 might be on Deno, that is to say, porting Meteor’s style of development and experience to Deno land. There is a reason why Ryan (author of Node) created Deno from scratch, at this stage of Node maturity, it is almost impossible, and would could argue to even desirable, to fundamentally change Node (adding sandboxing or removing package.json, changing the APIs, etc.) without breaking the entire ecosystem, so people are fine living with Node legacy and accepting the tradeoffs, I think the same can be said about Fibers and todays Meteor.

The culprit is exactly that. meteor is stuck like node.

But a new “meteor 2.0” would neither be compatible with meteor and would probably have very little to do with it (and would require funding, etc.). Personally i also think there are enough alternatives, but the simplicity of meteor is still unmatched and i would love to see a spiritual successor.

[alawi]

@macrozone

The culprit is exactly that. meteor is stuck like node.

Yup, that sums up my thinking, another way of stating my pragmatic argument.

@macrozone

Personally i also think there are enough alternatives, but the simplicity of meteor is still unmatched and i would love to see a spiritual successor.

I like that spirit, I’m open to trying to get Meteor spiritual successor on Deno and I’m willing to collaborate with anyone who is onboard with that vision. But yeah I think the NodeJS ecosystem is maturing, stabilizing and getting saturated already.

And I personally won’t choose anything other than Meteor on Node. It is battle-tested, has very well understood behaviour, has tons of packages and it glues a lot of stuff together nicely while maintaining backward compatibility. I evaluated every NodeJS framework out there and it is either DIY webpack, express, graphql, passport and 50+ other packages, or some legacy MVC framework, or let us outsource/lock-in our entire server (aka serverless), or new shiny framework that wraps express, graphql, passport and the 50+ other packages but hey we now have Typescript as well.

Frankly, I think the entire NodeJS/NPM/JS ecosystem is a cluster mess (to put it politely) that needs refactoring and it is getting constantly patched, it came along way over the last decade, ES6 is just a Javascript language patch, Meteor managed to abstract a lot of that churn thankfully, and there are good/valid reasons for Deno creation.

[macrozone]

[quote=“alawi, post:12, topic:53539”] Like I said, we’ve to be pragmatic and opinionated and not try to morph Meteor to something that its not.

I think we should keep the discussion focused on what to do with Fibers in the short-term.
[/quote]

well, then there is only the way of maintaining fibers or as mentioned above find patterns that would not work with fibers, identifiying them and warn.

[zodern]

I don’t think maintaining Fibers ourselves is necessary or a good long term plan. According to the readme, he is willing to continue maintaining Fibers, but warns that someday an update to Node or v8 could make it impossible to use Fibers.

I’ll continue to support newer versions of nodejs as long as possible but v8 and nodejs are extraordinarily complex and dynamic platforms. It is inevitable that one day this library will abruptly stop working and no one will be able to do anything about it.

Meteor uses Fibers to implement async/await: https://github.com/meteor/promise
If you look at a server file that uses async/await in a built Meteor app, they’ve been replaced with Promise.async and Promise.await. MDG’s plan was to hide the use of Fibers over time with promises to simplify using Fibers and reduce the impact of removing them in the future.

As far as I know, there are 3 main uses of Fibers:

1. Meteor’s apis that use Fibers to allow writing synchronous code, for example cursor.fetch, HTTP.get, or Meteor.user. We could create a separate method that returns a promise, and over time encourage it and depreciate the old methods. Though as @macrozone mentioned there would be challenges with getting this to work with Tracker/Blaze. Maybe it is time to rethink how we write reactive code?
2. App code that expects to be in a Fiber and is called by Meteor, for example methods, publications, and Meteor.startup.
3. Some information is stored on the fiber’s object (accessible from Fibers.current). This is used for Meteor.userId(), by core packages to detect if running in a Method simulation, and Kadira/Monti APM use it as a lower overhead alternative to async hooks.

Depreciating the use of Fibers would probably be a very long and difficult process, but I think it is worth starting due to the warning given by the node-fibers maintainer.

[alawi]

Although this statement from Fibers github repo is worrying and is true:

I’ll continue to support newer versions of nodejs as long as possible but v8 and nodejs are extraordinarily complex and dynamic platforms. It is inevitable that one day this library will abruptly stop working and no one will be able to do anything about it.

Just read that sorry,

[alawi]

Yeah, after reading this statement I’m getting on the fence as well, I guess @macrozone has a strong point then. I mean if the author is claiming that it might get too complex to maintain that integration/library, then the long-term plan should be considered carefully.

Perhaps phasing out Fibers gradually needs a serious thought and long-term planning.

[alawi]

It looks like Fibers currently supports Node 14 which will be supported until 2023-04-30, around 2 years and 9 months from now (assuming no minor patch will break Fibers).

NodeJS v15 (2020-10-21) might give us a hint on how Fiber might behave in v16 which if it turns out to be compatible with Fibers might extend the usage by another 2.5 years. Perhaps it might also help if the Meteor core team can get in touch with library author to see if he can elaborate on his statement and check if he has any visibility on near-term potential challenges. Lots of credits and respect to the author for building and maintaining this library thus far.

After thinking more about it, and reading the author’s statement, I came to agree with rjdavid, zodern, macrozon and dr.dimitru I think it is worth considering the longer-term phasing out of the dependency on Fibers to ensure Meteor can stay up to date with the latest Node for the foreseeable future.

And as jkuester suggested that refactor might indeed be on a Meteor 2.0 branch, it might also be an opportunity to modernize the entire code base to ES6, move some stuff to NPM and add types where needed, this work can be done gradually and gracefully if planned long-term correctly, it is important by not urgent task. Furthermore, by starting this effort on the right direction (i.e Meteor 2.0 branch), the meteor team/community will surely give a very strong social signal on their intention to keep Meteor around for a very very long time. And I hope benjamn has to some interest in this task, as I’m sure he gave this effort a lot of thought, at least he can be consulted on it.

Every challenge brings new opportunities for growth and I think there is enough smart/skilled people around this system with strong incentives to pull this refactor off elegantly.

[storyteller]

What about this: https://github.com/chris-visser/ddp/tree/master/packages/client by @cloudspider

[alawi]

I’m a big fan of this library direction.

I personally think that’s the right architecture/path forward, and this is how I’m developing some of my Meteor apps, decoupling client from sever and offloading the caching and updates to each view layer framework. In the case of React for example, there is useQuery and SWR doing very close to what tracker/minimongo are doing.

This could potentially evolve to be the Meteor 2.0 client data layer, a thin client downloadable from NPM and pluggable to any front-end framework or client.

[cloudspider]

Its good to note that its pretty much a work in prigress package. I’m fully committed to it though.

[alawi]

If there is an agreement on this direction I think many will step in to help building it.

[cloudspider]

My take on building survivable open source projects is the same on my viewpoint for larger projects.

• Composability and layering as the highest priority

Fibers is not a problem, Minimongo is not a problem, DDP is not a problem. In fact, they are great. What makes them less great is that they can’t go without each-other in the current setup. All of these packages are tightly coupled. Removing one will destroy the other.

We need to start focusing on decoupling features, have the individual packages be cross-compatible with the entire ecosystem and leverage Meteor’s build tool with smart packages to create drop-in features for Meteor.

I’ve started the DDP package for that reason. To have something that allows me to optionally connect with other very popular framework and have the best of both Meteor and for example Nuxt and Next.

• Composable
• Layered (low level fast changing API vs higher level less changing till never changing API’s / facades)
• Sprinklable (Optional low level integration for Meteor into existing ecosystems and tools)
• AWESOME documentation for new and experienced devs

This is what makes Vue so great. It has been rewritten in the background without actually changing a lot to its high level facade.

[romared403]

by starting this effort on the right direction (i.e Meteor 2.0 branch), the meteor team/community will surely give a very strong social signal on their intention to keep Meteor around for a very very long time. And I hope benjamn has to some interest in this task

[macrozone]

The former maintainers of mdg worked full time on that, they got paid for that. It was their business. Tiny now took over this business.

The point of tiny acquiring meteor was to keep it alive for current users, to maintain galaxy, give support. I don’t think that they have the intention to do something new. A non-backwards compatible version 2 won’t be accepted by the community, not without a clear migration plan.

The only way out is to slowly reduce the usage of fibers. First in the core and core packages, afterwards i user-space, which is quite hard as explained above. It is not possible to remove fibers (co-routines) without breaking changes for everyone. So there is a long way until a breaking meteor 2.

Its probably easier to either just maintain fibers (its anyway unclear whether it will break at all) or to deprecated the whole pub/sub/collections data-layer (which has other flaws than fibers in my opinion).

[alawi]

@macrozone

The point of tiny acquiring meteor was to keep it alive for current users, to maintain galaxy, give support. I don’t think that they have the intention to do something new

To their credit, Tiny and the new Meteor team are way more communicative and in touch with the community than MDG so I don’t think we need to speculate too much on their position and direction.

@macrozone

The only way out is to slowly reduce the usage of fibers. First in the core and core packages, afterwards i user-space, which is quite hard as explained above. It is not possible to remove fibers (co-routines) without breaking changes for everyone .

@macrozone

to deprecated the whole pub/sub/collections data-laye

I agree and understand, that is why it is Meteor 2.0 it will be breaking change but given the author warning, I think it make sense to invest in that direction slowly, again it is not urgent but desirable direction. And frankly Meteor has been on this path for several years now (since they open up the view layers), although I do see a path technically of migrating existing classical Meteor apps, and I think it can be done gracefully, also put this in perspective, this discussion is long-term planning and due diligence, not to cause any additional FUD for newcomers etc.

@macrozone what is your position/recommendation on this? it is not clear to me, there are really two choices here as far as I can see (feel free to suggest others):

1. Meteor team (and whoever else happen to use Fibers) invest in maintaining Fibers after Node14, despite the authors warning and hope for good, while slowly minimizing the dependency on Fibers in case it broke and revisit this discussion again when it does break and Meteor needs catch up to latest Node, that is the pragmatic approach, and I think it is ok. I’ve a feeling that this library won’t break in any major way, but the author warning is concerning, that is why I suggested that the Meteor team reach out to him try to get more info.

2. Start the migration effort soon on a breaking Meteor 2.0 branch and use this an opportunity to modernize other aspect of the stack well

I think anyone who has been using open source (and specailly node/npm) for some time has been in similar situation. Again I see this as opportunity now and that is why I switched my view from maintaining to migrating, but I’m keen to see what others think.

[macrozone]

i wouldn’t start a meteor 2.0 branch. With typical semantic versioning, you would increase this number on a breaking change and its not the time for a breaking change.

i would only maintain fiber if and when it won’t work for any future node version.

instead someone should go to the whiteboard and sketch out how a fiber less pub/sub could work (giving up isomorphic code? working with promise in a smart way?) and provide a new api that does not rely on fiber without removing the old way of doing things, but try to deprecated it (and warn).

Personally i have only one bigger project that uses pub/sub anymore and tend to migrate to graphql in that project (if i need to migrate at all). I would advice to at least consider switching to another (existing) datalayer as an option.

[cloudspider]

Hence why I started a DDP library. Its client only at first, but maybe I should aim to build a server version as well. A bit like how the Apollo Server and Apollo Client work together

[romared403]

@romared403

by starting this effort on the right direction (i.e Meteor 2.0 branch), the meteor team/community will surely give a very strong social signal on their intention to keep Meteor around for a very very long time. And I hope benjamn has to some interest in this task Bluestacks Omegle

by starting this effort on the right direction (i.e Meteor 2.0 branch), the meteor team/community will surely give a very strong social signal on their intention to keep Meteor around for a very very long time. And I hope benjamn has to some interest in this task