How to migrate large projects to Async DB methods

3.0 migration
1

We have started migrating a relatively large project comprising six sub-projects (all Meteor) sharing code, including our DB methods. We have delayed this migration because it requires a lot of work. This is a very active project with business requirements being added and changed regularly, so we cannot do a code freeze to migrate. We were also waiting for the Meteor 3.0 APIs to stabilize (which happened with the RC releases).

The most significant work in our case for this migration is making the DB methods async due to the unfortunate effect of async/await bubbling up to the entire call tree. When I first tried this, one simple findOneAsync() call required changing 20+ functions for the first layer (I stopped and did not try converting the first layer of 20+ functions to async; I estimated I would have touched scores of functions/files to update this single DB method. This would be chaos with multiple developers and projects reusing these functions).

While figuring out the best way to go about this, I encountered a straightforward tool available in Meteor which will be the best tool for this migration: Promise.await()

How does the migration work while using Promise.await()? An example might explain this better:

Code before:

function getDocument(selector) {
  return Documents.findOne(selector);
}

function getRecordByDocument(selector) {
  const document = getDocument(selector);

  return Records.findOne({ docId: document._id });
}

Migrating only the getDocument() function and not migrating other affected functions

function async getDocument(selector) {
  return Documents.findOneAsync(selector);
}

function getRecordByDocument(selector) {
  const document = Promise.await(getDocument(selector));

  return Records.findOne({ docId: document._id });
}

Using Promise.await(), we can select which functions to migrate. Our developers doing actual migrations can choose to migrate as many functions as possible. Those coding for business requirements can choose to migrate the affected function without being forced to migrate the entire call tree and other related functions.

Later, we can also migrate those Promise.await() by function.

P.S. We also override our Meteor.callAsync() in the server and wrap all calls with Promise.await(), which is very helpful in keeping our SSR implementation synchronous while converting the entire DB methods to async.

Squeezing the last benefits of fibers in Meteor (even during the migration out of fibers itself).

6 Likes
2

That is very clever! :smiley:

Had we known… :sob:

But we made it through ourselves anyhow. Great hack though!

What we did:

  • if we had a function we wanted to convert to ASYNC, but we weren’t committed to changing & thus converting callers everywhere at once:

  • we “duplicated the function, according to the Meteor DB functions *Async schema”:

Eg. we’d have, for a time, both these functions:

function getDocument(selector) {
  return Documents.findOne(selector);
}

function async getDocumentAsync(selector) {
  return Documents.findOneAsync(selector);
}

And then we could convert the calling code piece-by-piece by using the new *Async variant, until the last Sync version had been replaced. Then it’d be possible to rename by global search + replace the *Async function to the original functions’ name in a later step.

Maybe it’d be a good idea to collect these patterns somewhere! :slight_smile:

1 Like
3

We are making the same changes in our project, in some cases it is difficult to update all parts where a function is used. For example, we first started with our actions. We use Promise.await wherever actions are used. In some cases, if the section where the function is used contains too many dependencies, we rename the function as …Async.

1 Like
4

We can combine these ideas

function getDocument(selector) {
  return Promise.await(getDocumentAsync(selector));
}

I can imagine this to be useful especially if you have “large” functions that you don’t want to duplicate the code.

And thinking more about it now, we can write a codemod for the entire project with these combined ideas. :bulb:

5

That could be very helpful alright. We currently use a fair bit of Promise.await to stop the annoying bubbling up when calling async node libraries. I see some mention of fibers in the code though. What’s the status of the meteor/promise in meteor 3.0, does it just work ? If so then I think this is a really smart way to move forward and should be easily doable with a codemod.

6

Promise.await() will also stop working in Meteor 3.0. The plan is also to migrate it later when you have the time.

I used these combined ideas now and realized another benefit: we can start implementing strict eslint rules (cannot be disabled) related to the migration except the rule for not using Promise.await().

So there can be a progression on using the eslint rules for this migration:
Stage 1: implement all eslint rules but allow to disable
Stage 1.1: try rules and adjust as new cases are encountered
Stage 1.2: migrate db methods one by one
Stage 2: implement all rules and do not allow to be disabled except Promise.await()
Stage 2.1: migrate methods using Promise.await() one by one
Stage 3: disallow Promise.await()

1 Like
7

Promise.await() will also stop working in Meteor 3.0. The plan is also to migrate it later when you have the time.

Right, of course it would have been too good to be true!! :rofl: :see_no_evil:

1 Like
8

Just a quick update here.

Promise.await() has helped us move slowly to async DB methods without forcing developers to worry about the async methods bubbling up the call tree. We are now also slowly handling those async methods without Promise.await() and typescript-eslint rules for promises are helping us catch those new promises that are left un-awaited.

We have several custom eslint rules that helped us in this migration, but eslint has a limitation regarding file boundaries. That’s where typescript-eslint helped to identify imported promises and ensure that we are handling them correctly