How did I migrate our app to new Async API (Meteor 2.8+)

announce
1

Our app was Meteor 2.7 and we decided to update it to 2.8 and migrate to new Async APIs.
At first, I thought I will go through function to function, component to component (we use React in front-end), but @mikkelking suggessed that I should consider to use some tools like codemod or jscodeshift. At this moment I agreed that’s brilliant idea. Thanks Mike.

I decided to learn jscodeshift and it’s interesting that jscodeshift wraps around recast library which benjamn (who is the most contributor of Meteor project) knows very well :slight_smile:

You can find the transform scripts I wrote here: GitHub - minhna/meteor-async-migration

At first I thought It’s just some function renaming. But then I realized that once you made a single change, you Ignite a chain of changes. The function which contains the await expression became async function. Then you need to search for all places where that function was used, then add await and async and search … It depends on how your codes are but onc small change can lead to few dozens changes in other places.
But the problem is not how many changes you need to do, you may have time to do that. The real problem is what if you missed some?

The other problem is, while I’m working to migrate the app, other people were still working on other features. That means I need to make sure those new features/modifications will also works. It’s impossible if I do it manually or I need to ask people to stop working for months which is also imposible.

With those tools, I can do it faster and I can merge new codes and run on them.
But how do I know if the migration works? Fortunately we have integration tests (Mocha) and e2e tests (cypress).
For the integration tests, we use dburles:factory package to generate test data, which now support async APIs.
We also use johanbrook:publication-collector package to test our publications. Unfortunately, it hasn’t support async publish function yet. I created a pull request but it looks like the package is abandoned. If you use this package, you can use my version at: GitHub - minhna/meteor-publication-collector: Test a Meteor publication by collecting its output.

This is the pull request for the migrating task.
Screenshot from 2022-12-24 23-44-51

Hopfully the tool can help someone migrating the existing Meteor app. Meteor is awesome. I love it.

Merry Christmas and Happy New Year.

16 Likes
2

Hi @minhna this is great.

Thanks for sharing.

I think we can optimize a lot of work on these migrations script if we centralize everything in the same place and share with all, like you are doing.

2 Likes
3

Thank you for sharing this, @minhna! We’ve been slowly phasing out Fiber-based Mongo API using a custom ESLint warning; this transform sounds really helpful for fixing a bunch of the issues at once. But yes, the “chain of asynchronicity” issue makes it really challenging since sometimes changing one function’s signature can have a huge ripple effect throughout the code.

2 Likes
4

Do it manually is like :joy:
75o992

2 Likes
5

We have a similar plan. This allows a team to do this step-by-step and allowing the team to still continue with business requirements.

One simple rule: fix only if you encounter in the page you are working with (tough luck if it cascades to a lot of changes with many calls to the function). We don’t allow this rule to be bypassed using eslint comments.

Then to those explicitly in charge of working on this, there is an option to run the eslint tool for this rule and see which files require fixing. Then work as much as he wanted, then commit.

Too bad that the eslint rules for this are too customized to the codebase. Especially that we are very strict that this rule cannot be bypassed without fixing.

6

Too bad that the eslint rules for this are too customized to the codebase. Especially that we are very strict that this rule cannot be bypassed without fixing.

We don’t mind sharing the gist of our rule, anyway, which I think is generic enough to be generally applicable: We just look for expressions like db.someCollection.update and flag that they should be db.someCollection.updateAsync instead. Definitely a very blunt detection mechanism that can miss a lot, though.

7

That was also our initial approach. But since we decided that we cannot disable the rule using eslint comments, we need to handle cases like objects that have update, insert , remove which surprisingly a lot of npm packages out there and even our own non-collection objects.

8

@rjdavid - Yeah, that makes sense.

We specifically always call our Mongo driver methods in the format db.collectionName.update (saving the collection handles onto a db object as a form of pseudo-namespacing), which makes it a lot easier to tell the difference between db.collectionName.update vs someOtherObject.update. But I recognize that this is an assumption we’re making based on how we define our collections, so it might not work in other codebases.

9

You’re right, in our codebase, we don’t use db.collectionName. syntax, we always use exported collection variables. In my script, I can open the exported file, look for the variable, if it came from new MongoCollection() init then it is a mongo collection.
But it won’t work if you do something weird likes changing the variable name before export, etc.

10

Would it be possible to use top-level await so as to avoid making the function which contains the await expression become an async function?

11

I don’t get it. Can you give an example?

12

I haven’t tried it yet – and I may not know how it works it.

My understanding is that node.js v14 supports top-level await, which I think is the ability to use await outside of an async function.

Here’s a reference. I’m probably missing key requirements.

13

I don’t think you can move everything out of function. How can you pass it the variable?

14

meteor wraps all packages (and the app code) in a function, so the ability to use top level await meaningfully depends on meteor, not node

1 Like
15

So top-level await cannot run in a non-async function? I thought that was the whole point of it. I had a feeling there was something I was missing about it.

16

top level await in node can run outside of a function, but top level await in meteor cannot without changes to meteor - because ultimately this code:

await whatever();

gets converted to something like

(function() {
   await whatever();
})()

I believe Meteor needs to change that wrapper to be async function then change it’s boot-loader to Promise.await it. Things get a bit muddier since meteor also converts await to Promise.await

1 Like
17

Top level await is coming in Meteor 3. It currently is in development at Top Level Await by zodern · Pull Request #12310 · meteor/meteor · GitHub.

I believe Meteor needs to change that wrapper to be async function then change it’s boot-loader to Promise.await it. Things get a bit muddier since meteor also converts await to Promise.await

We were initially planning to release it in a Meteor 2.x version, but supporting top level fiber code in a backwards compatible way, and trying to create a spec compliant implementation of top level await that uses Promise.await seemed to be incompatible, so we are releasing it in Meteor 3 which uses native async/await instead of Promise.await. With enough time and effort I think we could have gotten it to work, but it doesn’t seem worth it with the latest plan for removing fibers.

Top level await only enables using await outside of any function. If you use await inside a function, the requirement for the function to be async still exists.

5 Likes
18

@zodern, once TLA is available in Meteor, does it mean Meteor will also support ES Modules? Or was this just syntactic sugar to mimic TLA?

19

It isn’t native top level await. Like ES Modules, it is implemented in reify. The PR for that part is at Top level await by zodern · Pull Request #4 · meteor/reify · GitHub. It tries to be spec compliant. When testing how compliant it is for the timing and order it runs modules, it is equivalent to webpack and v8, and significantly better than the other bundlers I’ve tested.

Meteor will still need to support environments for a while where using native module isn’t possible or has downsides compared to using reify. Next year I want to see if there is someplace we could start using them - maybe the meteor tool, or build plugins, or at least parts of the server.

3 Likes
20

For anyone interested: I’ve set up a small repo with the necessary files to have ESlint help you with tracking down code that should be updated.

You can find it here: GitHub - cluelesscamel/meteor-fiber-free-diet-eslint: Custom ESLint rules to help track down non-Async meteor code

7 Likes