Handling database migrations in Meteor

announce
1

So, I know a million other things are going on right now, but some topics never go stale. One of them is database migrations. And as at Vazco we’re working on multiple projects, we’ve decided to write about some of our “migration schemes”. Here’s the first one: “On Database Migrations in MongoDB” (it’s also available on Vazco’s blog). It’ll take us some time to describe others, so feel free to comment on this one in the meantime. We’re also working on more content (not necessarily Meteor-specific), so stay tuned!

(I feel proud enough not to be ashamed of spreading the word on r/programming and cross posting it to r/mongodb and r/meteor.)

13 Likes
2

Just throwing in some simple migration code which serves well on a couple of mid-size projects. The code example is from a project that has 16 migrations since its start 2 years ago. We only have a hand-crafted schema which we use for doing the same validations on the client and server side. I deleted all but the first migration as an example.

import { Meteor } from 'meteor/meteor'

// Migrate the database
// The array `migrate holds an object for each migration job with the following properties:
//   no (int) - A job number that needs to be incremented manually
//   label - A job description
//   up (fn) - An asynchronous function that performs the actual migration, it should return something meaningful for logging

const Migrations = new Mongo.Collection('Migrations')
const FIELD_DELIMITER = '~'
const tenantId = Tenants.getCurrentId()

const migrations = [
  { no: 1, label: '#127 Rename payment to paymentType and account to accountType in Sponsorships', up: async () => {
    return await Sponsorships.rawCollection().updateMany({}, { $rename: { payment: 'paymentType', account: 'accountType' } })
  } },

]

const migrate = async () => {
  const lastMigration = Migrations.findOne({}, { sort: { lastNo: -1 } })
  const lastNo = lastMigration?.lastNo || 0
  const migrationsToApply = migrations.filter((m) => m.no > lastNo).sort((a,b) => a.no > b.no)
  try {
    migrationsToApply.forEach(async migration => {
      LOG.info(`Start migration ${migration.no}`)
      const result = await migration.up()
      LOG.info(`Migration '${migration.label || migration.no}' (${migration.no}) done, ${JSON.stringify(result)}`)
      Migrations.upsert({}, { $set: { lastNo: migration.no, lastMigration: new Date() } })
    })
  } catch (error) {
    error && LOG.error(error)
  }
}

Meteor.startup(async () => {
  if (Meteor.isTest || process.env.npm_package_scripts_test) return
  await migrate()
})
3 Likes
3

Nice article @radekmie. It’s a tricky subject migrations :sweat_smile: so thanks for writing it.

One of the most interesting thoughts in your article for me was right at the bottom - using transactions. I had never thought of that but that could make it a lot easier to roll back from errors. For us one of the most difficult elements of migrations is when and how to run then. We didn’t like running them on server startup which tends to be the default (as posted also by @softwarerero) so we added a small UI to trigger them when we wanted. Then we had the issue of users still using the system when we wanted to run them so we added a maintenance mode to be able to kick them out. Even with all this though we still have the odd migration where a doc gets added/modified with the old schema while the migration is happening. I’ll look into transactions more and see if they can help.

BTW - We’ve gone through 181 in 6 years :laughing:

1 Like
4

Understandable. I do agree that it’s much easier to do it on startup, though – otherwise you have to take care of the code, as it has to be able to work with both versions of the data

That’s still a lot! Go and comment in here that you’re also working on a Meteor app for a couple of years. I think most people think Meteor died already :expressionless:

5

Thanks for sharing this article @radekmie
I’ll definitely read this later :raised_hands:t2:

6

Agreed… if your app is only running on a single server!

If you’re working with multiple containers/servers, and you’re deploying with Galaxy/Meteor Cloud, then you kinda have to make sure your code works with both versions of the data anyway, since there is always some gap where both versions are live as you replace the old version with the new one.

(I guess you could avoid that with multiple servers by having scheduled maintenance/downtime where no servers are running as you spin up the new version.)

7

That’s also covered in the text :stuck_out_tongue:

8

Yep, you definitely mentioned it! I promise I read the article (which was great btw – thanks for sharing it!) :smile:

Just wanted to highlight this for other readers, since it may not be obvious that running on startup doesn’t guarantee data consistency if you have multiple servers.

2 Likes