Get notified when your package breaks on new Meteor versions

packages
1

… that’s what @sashko and I want to make happen but we need your help!

We want to make sure that packages get tested against pre-release versions of Meteor so kinks get worked out before people upgrade. The idea is that package authors can register to have their test-suite automatically run whenever a pre-release version of Meteor is announced.

It seems like repurposing a CI tool like TravisCI or CircleCI would meet our needs. Can anyone in the community suggest how to go about doing that? Looking for as simple a system as possible.

If this is something that can automatically be done through Github + TravisCI integration, that would be ideal!

Please help!

2 Likes
2

I think we should run the tests for a certain community package on 2 conditions:

  1. A commit or release in the package
  2. Nightly

Each time, they should run using the Meteor devel (most recent commits) and master (most recent release) branches.

3

I believe @alanning is asking about something different. Please recheck.

@alanning - Great idea :smile:

Regards,
Sanjay Kumar

4

I’m pretty sure it’s the same thing… What do you mean?

5

here is my understanding - Please correct me if I am wrong -

@alanning is talking about automatically test the packages when there is a new release of Meteor. The Package owners will get notified and they’ll update their code if the package has any issue with latest Meteor version.

What you are talking about is testing the Packages when there is a new version of the package + testing the packages on periodically basis.

Again, please excuse me, if I am missing anything and correct me if this is the case.

Regards,
Sanjay Kumar

6

@deligence1, Thanks for the summary. Yes, you are right in both cases. But because @sashko is also proposing the use of Meteor devel for tests, his case is a superset of my original suggestion.

New Meteor releases all start life on the devel branch so we don’t have to wait for the actual pre-release. Basically we’d know after the next nightly test run if a commit to Meteor core caused issues for our packages.

Do you happen to have experience with CI tools? If we had to host our own community CI instance, I think having people add some kind of file to their repo like we do for TravisCI would be acceptable.

I’ll ping the Velocity maintainers too since they probably have experience with this kind of thing.

7

Yeah I think just having easy directions for people to set this up on Travis would be enough. Then we at mdg can make a dashboard of everyone’s Travis badge!

8

OK I looked at this for a couple minutes, and it doesn’t look like periodic/nightly tests are a thing on any of the popular build services. However, I think any solution that doesn’t use one of the easy-to-setup popular services like Travis, Circle, or Codeship is a non-starter - we definitely don’t all want to get into the business of writing CI servers.

Anyone have any ideas how we can use one of these services for this? A simple starter solution would just be to run these tests when there is a new commit in the package, in which case you wouldn’t know if a new release of Meteor is breaking your package unless you make a commit. Then perhaps we can figure out how to run the tests more often?

9

Thanks @alanning. Thanks for making it clear to me.

No, I don’t have experience in CI tools :frowning:

Regards,
Sanjay Kumar

10

Validation and Integration testing covers a lot of this kind of work, and is a big focus of the StarryNight utility.

Long story short, a reference app is needed to baseline the packages. (Which is part of the reason we were dismayed when the Parties example was deprecated, because it no longer provided a baseline for diff-testing D3 data visualizations during upgrades.)

Here’s an example of the CKCC reference app we’re currently piecing together to track the clinical:Meteor packages. You can see it has the build badge.

CKCC Reference App

And you can find the Travis build here:
https://travis-ci.org/awatson1978/ckcc/builds/86129742

And the build history here:
https://travis-ci.org/awatson1978/ckcc/builds

After you have a reference app with all the packages you’re keeping track of, it’s as simple as creating a new git branch of the app, upgrading to the new Meteor release, and running the test suites. Here’s what using Travis and the GitHub Status API together looks like, when you do an upgrade of core packages and files:

Pull Request #79 - Merge Develop into Master

The .travis.yml file we’re using is a little more complicated than the ejPSng file that’s the travis-ci-meteor-packages and it’s associated documentation reference. Ours isn’t just launching TinyTest to run unit/verification tests. It’s also doing things like fetching the packages, scanning them for validation test commands, building the validation script from them, and then running the validation suite.

Currently we only have 500 tests out of 5000 implemented for this particular application. I’ve worked on a few other Meteor apps where we had many thousands of tests like this; but this is the first one that’s been able to be open sourced. We have the 2000 tests from MedBookCRFs, 2500 from ClinicalTrials, and the 500 that are currently implemented. But we should have 5k tests running across a few dozen packages by the end of the year.

The advantage of using Validation tests, of course, is that they can walk across technologies when you swap them out and replace them. Rewrote your front-end in React? The validation tests should still work, and allow you to do the rewrite with TDD practices. Same with upgrading the core APIs. Validation tests are able to walk across underlying package upgrades and changes, assuming the upgrades don’t actually break anything.

Anyhow, we started working on a suite of reference apps, clinical demos, and an associated page of packages for clinical meteor. As you can see, it’s very much a work-in-progress. This stuff takes a long time to assemble and put together. But we know where we’re going, and how to get there.

But the general idea is twofold:

a) have tests running on a per-package basis with TinyTest or SpaceJam.
b) have reference apps running integration/validation tests across multiple packages and in a real app environment using Nightwatch/StarryNight or some other Selenium launcher.

We’re currently looking closely at practicalmeteor:mocha for our package verification tests; and StarryNight/Nightwatch for our reference app validation tests. We’ve also set up a repository at http://github.com/clinical-meteor to keep packages that are under Travis integration.

But yeah. When I was an Oracle Admin with NY Presbyterian Healthcare System, and working with Cerner EMR, we had banks and banks of these tests that we ran on Daily, Weekly, Monthly, and Seasonal basis. There’s a testing theorem in systems engineering that you can only test one change at a time. It’s a Six Sigma thing. So we would queue up months of changes at a time. Pharmacy upgrading their robot? Scheduled for integration testing three weeks from now. Lab upgrading their barcode scanners? The following week. Patient portal going online? The week after. And we would run those banks of tests against our reference apps again, and again, and again, and again…

11

Do you know of any generally-available CI services that would be easy for people to set up? It’s not hard to do it on custom infrastructure like Jenkins, but that’s not an option for most people.

12

Travis and Circle are the two that are most talked about. They both support the GitHub Status API and offer open-source tiers.

13

The thing that I’m not sure how to do is to trigger a test build on those CI services when a new Meteor pre-release has been created. I think the following would work (maybe there is an easier was):

Create a web service that uses the GitHub webhooks to get notified, when a new tag is created in the meteor/meteor repository. When this happens, it uses the API of the CI service, to trigger a new build with the new Meteor version. It can pass the Meteor tag via the API.

With CircleCI this is supported: https://circleci.com/docs/parameterized-builds. Travis and Codeship don’t have such an API. They only have a restart build API.

14

For packages that use Velocity, we need to merge the additions into meteor/meteor first or MDG need to release the velocity:meteor release too, because we currently have the Meteor fork with the Velocity additions.

1 Like
15

Some folks have written code to call TravisCI’s restart build API nightly.

You can read the full issue thread here: https://github.com/travis-ci/travis-ci/issues/582

Some of the solutions mentioned in the thread:

HTH
Blake

1 Like
16

Here’s an :bulb:

Amazon Lambda has a scheduling function. It would be trivial to create a node script that is called regularly that would check the versions of Meteor and kick off a job on a given CI service. You could also make the script look at a file in a git repo that contains all the badges (insert CI service here) and the script would go to that job and kick it off.

1 Like
17

Unfortunately, to kick off a job on, for example, Travis CI, you need the repository owner’s GitHub API key. @evanyou had a great idea that we could instead open a fake PR on that repo, and then the tests can be set up to run on a PR.

1 Like
18

To expand on this idea, we could just have one long-running open PR called “Meteor test status” and just push new commits there; the new commits would cause the tests to get re-run.

19

would the push to the PR be a manual or automated task?

20

@sam that would be automated in our CI jobs.