Migration to 2.8 Async version

help
21

I would recommend something more collaborative than a discussion to avoid the core team being bottlenecks.
Maybe a shared google doc where people can contribute? Or does this board thing or github have something like a wiki page?

22

Everything should be async.

23

Iā€™m afraid a google doc can be too messyā€¦ At least on Github, it looks easier to chime in, like in this discussion.

BUT we can find a better place if the community doesnā€™t like it there, not a problem.

EDIT: I think that instead of a discussion I can create a new MD file in the Meteor repository, like this one, so everyone can collaborate. Then people can easily access it here in our guide. What do you think?

1 Like
24

Idk - at this time I feel we need something highly collaborative without any approval workflow, like a wiki page

25

Thanks @denyhs,

I think that forcing the switch to asynchronous logic on the client side as well will bring an ultimately unnecessary amount of conversion work. I fully understand the motivations, but letā€™s keep the legacy code in mind. In the Blaze environment, all the reactivity contexts should be reviewed: helpers, autoruns, dependencies and the associated call chains; basically, the entire code and the associated tests should be completely revised. To date, there is not even the possibility of doing this as Blaze does not yet support this model. Then, it would only be a matter of bringing mimic asynchronous with a Promise.resolve, I donā€™t think you want to bring minimongo to a native asynchronous version. I can assure you that, for applications like ours with thousands of lines of code, it is impressive work, bordering on a complete rewrite of the code.

Already the rewriting of the asynchronous server context is forcing us to distinguish the code between server and client in order to make up for the Trackerā€™s shortcomings, we would then have to revise everything to bring it back to its initial state, doesnā€™t that seem a bit too complex a migration process?

When I was talking about a possible underestimation of the client side (see here), I was referring to these outstanding points, it seems to me that we are still working on sight.

2 Likes
26

I completely agree with @pbeato.

The client-side source conversion would not be an adaptation but a complete rewrite of the source with all the consequences of the case and moreover without a real reason.

In fact, for a new application designed from scratch it would certainly be cleaner to have the async code also on the client side but, since unlike the server (Fiber and Nodeā€¦) there is no intrinsic reason to have everything async, I think that as far as the client is concerned, guaranteeing both ways (sync and async) could be the wisest choice to avoid that many applications, having to be completely rewritten, end up abandoning Meteor completely.

I urge you to consider this possibility.

2 Likes
27

Considering that we already donā€™t have Fibers on the client, it might be possible to keep Mongo sync in the client.

Iā€™ll need to confirm if this would be possible.

Maybe @radekmie could already confirm?

EDIT: So yeah, apparently we should be able to keep both actually. The sync version should continue to work. But in the future, weā€™ll need to worry about renaming the API or not (client side). We could keep the method updateAsync, or rename the current sync one to updateSync. But this is something we can all decide when the moment arrives.

3 Likes
28

I think @rtrevisanā€™s reasons are absolutely valid.
As far as our application (meteor+blaze) is concerned, a lot of effort will be required for server-side porting. Since the client could be kept synchronous, I would expect not to be forced to modify it, at least in a first step

29

Yeap, thatā€™s the idea. We want to do this avoiding forcing things that donā€™t need to be forced.

2 Likes
30

I think it should be decided now, let me try an example:

Suppose we have a simple function now shared between client and server and used, client-side in a Blaze helpers or autorun and server-side in a method:

function hasPrivilege(prv) {
return userPrivileges.findOne( { _id: Meteor.user()._id, privilege: prv })
}

On the server side I would have to rewrite it asynchronously and change the whole call chain to support the new signature:

async function getPrivilege(prv) {
return await userPrivilegesAsync.findOne( { _id: (await Meteor.userAsync())._id, privilege: prv })
}

While on the client side I would have to keep the function synchronous, losing the isomorphic nature of Meteor that allowed client/server code sharing.

When reverting to the standard nomenclature, things are reversed:

Server-side

async function hasPrivilege(prv) {
return await userPrivileges.findOne( { _id: (await Meteor.user())._id, privilege: prv })
}

While on the client side we have

function hasPrivilege(prv) {
return userPrivileges.findOneSync( { _id: Meteor.userSync()._id, privilege: prv })
}

A migration that becomes really complicated and with a high probability of writing unnecessary chunks of code.

There must be other ways of doing this that I canā€™t see at the moment.

I would also like to know the opinion of those who are in charge of the product in order to reassure us users about the overview.

31

I still think that a flag and a Promise for observe and observeChanges would be enough, but I didnā€™t got time to experiment with it yet.

I wrote it a couple of times, but I canā€™t find it now. Itā€™s true, that we can keep the sync APIs on the client, but the question is how does it affect the isomorphic code (i.e., server-side). One solution is to have them on the server but throw an error immediately, second is to make them no-op (e.g., fetchSync would return an empty array, updateSync wouldnā€™t do anything), third is not add them at all. All have some solid pros and cons.

At the same time, nothing stops you from renaming the functions in the app, soā€¦

32

As I understand, the only way to make codes work on both client and server without separating client/server codes is using async api on both sides.
e.g:

const links = await Links.find({}).fetchAsync();

If we have sync api on client and async on server, it would be like this:

let links;
if (Meteos.isClient) {
  links = Links.find({}).fetch();
} else {
  links = await Links.find({}).fetchAsync();
}

So the function which has those code must be async as well. Then it doesnā€™t make sense to use the sync version in share code.

So have a version of sync api on the client-side: updateSync, fetchSync could be better.
On the server-side, it should throw error because ā€œdoes nothingā€ is dangerous, itā€™s hard to find where the problem is.

33

Even if we consider rewriting everything asynchronously, the change to support client-side responsiveness in asynchronous contexts, see this PR, still requires the code to be split between client and server for it to work reactively. If someone can give an example to better understand the approach we need to take in migration.

function hasPrivilege(prv) {
const {_id} = Meteor.user;
return userPrivileges.findOne( { _id, privilege: prv })
}

(I used Meteor.user() and not Meteor.userId() just to have two asynchronous functions and it is just a meta code for an example))

Would it have to look like this to be compliant?

function async hasPrivilege(prv, computational) {
const {_id} = await Tracker.withComputational( async() => await Meteor.userAsync(), {computational} );
return await Tracker.withComputational(() => await userPrivileges.findOneAsync( { _id}, privilege: prv }, {computational))
}

Next we will see how to make this understood by Blaze now still synchronous.

34

Can we look at the Meteor design patterns as they are taught and applied as a framework to steer the roadmap ?

  • Server side code (typically found in /server ): becomes async only to enable node upgrade
  • Pure client side (typically found in /client) : the blaze (&tracker) /react/ā€¦ code that relies on the synchronous patterns of the packages remain synchronous except where they interact with the server : Meteor.callAsync, this change alone has already some serious impact on refactoring clients side code
  • shared code (typically found in /api) : can we make the client stubs explicit (now they are implicit) and continue to use sync code on stub side ? Then we still have to find a solution for code that e.g. used in SimpleSchema and thatā€™s run on both client and server side
    Focus should go to allow node migration asap and keep re-factoring managable and predictable.
2 Likes
35

@polygonwood I agree with you, just two clarifications:

The modification of Meteor.callAsync is not so impactful because Meteor.call was already treated as asynchronous being the response handled in a callback.

The shared code can be asynchronous since the method stub already provided for this.

Pure client code should remain synchronous.

This scenario at least for version 3, then we can think about upsetting this, but with the security that server-side compatibility with the latest versions of node is guaranteed.

36

Do you expect it to be released with version 3 or 2.10?

37

Based on feedback here and some discussions weā€™ve had, we plan to:

  • On client-side, for versions, 2.x and 3.x, we will keep both synchronous and asynchronous options. You can use, for example, findOne() to keep your code synchronous or findOneAsync() to make an asynchronous call on the client-side.
  • On server-side, up to version 2.x we will have both options. You can use findOne() to keep your code synchronous (with fibers) or findOneAsync() to make an asynchronous call. In version 3 they all become asynchronous. We keep both methods valid, but they will behave the same way. Both findOne() and findOneAsync() will be asynchronous and without fibers.

Let us know if there are still concerns about this approach.

2 Likes
38

Instead of splitting by the client and server, another option is to instead split it by Minimongo collections that are in memory and remote collections stored in Minimongo. In memory collections can be used on the server, and like on the client they can still work synchronously.
Iā€™m not very familiar with SSR, but Iā€™m wondering if it would need to use in-memory collections on the server synchronously.

39

Please note that Tracker.withComputation is also available on the server and null is a valid computation.

No ETA for that, sorry.

Iā€™m a little bit worried about that, as then the return type of findOne becomes different on the client and server. If an app relies on findOne returning an object, it wonā€™t work anyway, therefore Iā€™d rather make it throw on server instead.

Thatā€™s why I suggested calling it *Local, e.g., findOneLocal so that itā€™s always sync and available on both client and server. The only difference is that itā€™s always ā€œemptyā€ on the server. (But nothing stops you from having a server-side cache thatā€™d populate it there.)

40

@zodern Could you better explain the distinction between the two types of collection, please? We use:

  • Mongo collections saved on the physical db and sent to the client by publication
  • minimongo collections in memory fed by publication
  • minimongo collections live only on the client

Mainly the reactivity is based on the first two contexts. Here we should have client-side synchronous access for compatibility with existing packages and Blaze. For the server and for shared code the asynchronous version gives us no compatibility problems.

This I had guessed, but the code seems to me to become a little less clean by adding an additional wrapper which has no benefit on the server. But I understand the difficulty in tracing the cotest in the client.

With the appropriate nomenclature and context, the solution proposed by @fredmaiaarantes seems good to me.

1 Like