Continuous Integration with Meteor, CircleCI, and Gagarin

Holy smokes, Batman! The game has been changed. True integration testing for Subscriptions/Publications (highlight from @arunoda’s hackpad):

describe("simple gagarin test on the client", function() {
  var app = meteor({flavor: "fiber"})
  var client = ddp(app, {flavor: "fiber"});

  it("should be passed", function() {
    app.execute(function() {
      // create posts collection and add a post
      var Posts = new Mongo.Collection('posts');
      Posts.insert({name: "firstOne"});

      Meteor.publish('getPosts', function() {
        return Posts.find();

        addPost: function() {

    // subscribe to getPosts publication and wait for the ready message
    var posts = client.collection("posts");

    // add a new post'addPost');
    // wait until new data comes to the client

    // check the new data arrived or not
    posts = client.collection("posts");

Depends on what you mean by an integration test. In principle, yes, TinyTest is sufficient… but in practice, if you have an actual matrix of integration tests, it requires creating a bank of reference apps and/or scripting the meteor tool and a CI server to create an autobuild system.

Thanks! That sounds like a pain :smile:

I’m basically thinking of using it for testing models (mutation on client and server via meteor methods) and other things that might hit the database… most other things in my current systems can be pretty easily unit tested.

Sounds like Gagarin might be the ticket. I’m so tired of Velocity breaking every upgrade… though it’s state should be ‘stable’ for some time now :laughing:


I worked with Gagarin a while back, I enjoyed it. It was very intuitive to setup pre-conditions, which is sometimes the hardest part of testing. By ‘pre-conditions’ I mean something like : ‘user is logged in, user is on the home page, user has one friend request’.
Also it’s really fun to watch the tests happen, your entire application functionality you get to watch go by in a flash, if something goes wrong you can take a screenshot of the error. I’m looking forward to picking it back up again soon. @samcorcos great article.

1 Like

@apendua I wonder if the creator of Gagarin has any comments to add?? This has so much potential.

Hey Guys,

@faceyspacey thanks for calling me out! And thanks to @samcorcos for bringing Gagarin to everyone’s notice.

Gagarin has always been intended as an integration tests framework. It is a totally independent test runner which can be thought as an alternative for velocity, but only in the integration & end-to-end tests area. It does not provide any decent support for unit testing.

For those who have not used Gagarin yet, the approach to writing tests is very similar to what Laika had to offer. So basically the test runner is responsible for spawning meteor processes and connecting to a webdriver instance. Then, it sequentially executes routines on those entities by sending source code through either webdriver protocol or a DDP method.

The code snippet brought here by @faceyspacey is a good example of what we can achieve being able to run any code on the server side (like with meteor shell). There are still some sharp corners, e.g. loosing information about the function scope when it’s send to the server/client, that we will have to polish at some point, but apart from that everything works surprisingly well.

Our biggest problem at this point - as some of you have pointed out - is the lack of good documentation, which I never had time to finish. So any help in this area would be most welcome. Ideally, we should build a website contaning documentation along with some interesting examples.

When we have it, we can compare the current state of the art with whatever expectations the users may have. Actually, I am quite sure that a lot of features you may be thinking about are already implement - they’re just not documented properly.

If you have any questions related to using Gagarin, don’t hesitate and ask them here! For anyone interested, a very good source of examples is the Gagarin’s test suite itself:



Well, how can we add “decent support for unit testing.” Unit testing after all is the easiest form of testing, considering all the challenges we’ve had with integration testing and to a lesser extent end to end testing. If it’s just a matter of adding Unit Testing to Gagarin for it to be the Ultimate solution for testing in the Meteor world, then we should make it a top priority.

More generally, what problems does Velocity solve that Gagarin does not?

How can I help?


I can help you to put some docs.
I did it here sometimes back:
(as I posted here)

May be shall we put this with the Meteor Guide.

@sashko any place for Gagarin in the Meteor Guide? May be in the future.


Yeah we’re still figuring out the testing story right now. We’re definitely taking a good look at Gagarin.


Hey Sashko, it would be great if Meteor developers were guided to utilize best practices. An out of the box model for writing and perform tests would raise the quality of Meteor developers, encourage their hire, and increase Meteor’s business adoption.


@arunoda I think it’s a great idea to use hackpad for scaffolding Gagarin guide. I think that creating a real documentation will require a much more structured work though.

@faceyspacey Let me state it this way: somewhere deep in my mind, there exists an ideal testing framework which integrates flawlessly with Meteor and is canonical and minimalistic in terms of the code you need to write to describe your testing scenarios. The problem is neither Velocity nor Gagarin is that “ideal” framework.

Velocity is really good in integrating your tests code with your application code (e.g. injecting code at build level) and it also provides a lot of freedom in terms of implementing custom frameworks on top of it. On the other hand Gagarin gives you much more control on your meteor test instances and simplifies communication between server and client because it’s run from outside the application itself. It’s not good for unit tests though because they tend to slow down dramatically.

By “they tend to slow down”, do you mean because of the startup time? I think we can reduce the startup time, if we would be able to connect to an already running instance of mongo, instead of booting up it’s own instance.

Besides that, one of the reasons we’ve implemented the glob pattern a while ago, is to be able to run only a selective set of tests. For example: gagarin packages/acl/tests/**.js -v to run only those in the “acl package”, or even specifying it further down.

Perhaps even further reduction in spawn time would be possible if gagarin had a watch flag, which spawns gagarin, but doesn’t end the process. Instead it should go watch the pattern for changes, and rerun those tests on change.

A big influence of gagarins startup time is the meteor build time though. And it seems to me that we cannot make that faster very easily. I’ve read about the performance gain of using webpack:webpack, (2 sec rebuildtime instead of the current 15+), but I haven’t tried it myself. However, if the ‘module patching’ they are advertising with really works, for both meteor and gagarin, then we can rerun all tests within a matter of seconds.

The ‘module patching’ that webpack is advertising with, combined with a filewatcher in gagarin, would be very powerfull.

1 Like

I think the information you hold when shared with the rest of us can help us all contribute to taking that ideal meteor testing framework from being “somewhere deep in [your] mind” to being a reality. Currently the information regarding making that ideal meteor testing framework and its challenges is only intimately known by a few (you, the guys behind Velocity, @arunoda, etc). I believe this dream needs to become a reality, and we need to make it a first class priority.

You’ve given a few abstract examples of the challenges faced here, but I think a conversation about the specifics (which already seems to be unfolding here) is in order if we ever plan to achieve this.

What does “injecting code at build level” actually mean (i.e. in depth)?

This seems to go hand in hand with “injecting code at build level.” It’s not efficient to build your application to run a unit test which can be more quickly run without being backed without your application–is that the idea? It takes gagarin on average like 30 seconds or something to build your app before your tests can run. Velocity has pulled off some tricks to do this faster by “injecting” code into an already built app? Is that the idea? It would be nice to do unit tests under the same hood as Gagarin, so you’re not using multiple tools for the same class of job.

It seems that the main issue isn’t even hot reloading of code, but rather patching state to the precise exact same “state” it needs to be to make your tests reproducible 100% of the time (along with any fixtures your tests may need). Cuz, we can hot reload code into the runtime pretty easily and basically in an instant for a small number of files changed (even with all the transpilation that goes on in build steps), but there is no guarantee that the application state will be exactly what it needs to be after running the first test, let alone after many tests. This is especially true given the opaque nature of closures, i.e. where state is hidden as private variables within closures. You could take snapshots of all of mini-mongo and put it back to where it was before each test, but it doesn’t mean other code won’t have changed private variables within closures. Would snapshotting and replacing the entire window object do the trick–I’m not sure, is there still artifacts lurking in the runtime?

On a side note, Redux through being unidirectional, functional, and immutable allows for such testing today (so long as you maintain its functional principles). For all apps to maintain these principles in all aspects of its code, it’s a hard pill to swallow, and definitely not possible for non-functional apps built already, and likely may never be fully possible even for new apps unless you wanna embrace the functional approach wholeheartedly. Even tools like React + Redux likely don’t get that write without being somewhat of a “leaky abstract”–something like Cycle JS would likely be a far better option (in pure js land). Even then, it comes at the loss of many of the good aspects of OOP, which–without making this the main topic for debate–is still the best tool for a great number of problems, and I would definitely say is an easier way to learn programming than functional programming (though, perhaps that’s like saying English is easier to learn than Chinese first because I learned English first). Regardless, OOP is definitely not going away and is still the predominant style of coding, so I think we should find a way to offer a lot of what Redux offers in terms of time traveling “debugging” today with traditional coding styles.

I’d like to point out what Tonic Dev ( ) does to quickly run any Node JS code you run in the browser–read this blog post by them: Basically, they use this somewhat new (or at least recently evolved) tool: to snapshot the entire NodeJS runtime, as well as its associated file system, at a lower level, similar to how VM containers are made. They can then roll back the entire application state flawlessly.

I’ve been looking into such things because I refuse to believe that the state of Velocity or the build times of gagarin will hold us back from the true benefits of TDD: basically immediate response and observability of your code. In short, with the current limitations the only benefits we are getting from tests is protection from regressions. While that’s important, I think the upgrade in development flow of true immediate TDD is in a world of its own. Check out Wallaby JS right now if you haven’t:

here’s the video tour:

As you can see, you basically have what Chris Granger’s Light Table was only effectively doing with ClojureScript (their glorified JS support never reached much more than a basic JS repl last time I checked). So Wallaby is doing similar stuff where it shows the results of your tests and inline errors within your tests and within the tested code AS YOU TYPE. We could only dream of such things without first getting our build time in order, or rather, workarounds where only changed code can both be patched, while syncing application state back to a previous snapshot.

Being able to code this way would get the full power out of tests. It would lead to a lot more tests, because more often people would start with them, rather than end with them just to catch regressions. In short, without such things TDD isn’t even a productive workflow for Meteor development today. It’s extra work that clients often don’t pay you to do, whose benefits aren’t immediately apparent.

Where my train of thought goes next is whether CRIU could snapshot the browser runtime (and replace it), not just Node JS on the server?? Because even if we were able to implement CRIU for server side testing, the client app still needs to be setup and torn down equally as quickly.

Now that, said, because of React, having to test your code on the client is becoming less and less important. The following project allows you to use JSDOM to render React on the server and test it. In addition, you don’t even need to JSDOM if you want to test React components individually. You can use React’s built-in testing addon. The problem with that though is that it only shallowly tests the output of your React render methods, i.e. for its precise return, not the return of nested child components. You need a DOM to do do that. What that means is if you change how you compose your components, but what’s written to the DOM is still the same, the test based on React’s testing utilities will fail, whereas the DOM-based tests won’t. So that’s where JSDOM essentially puts ReactTestUtils to shame. …So anyway, perhaps we don’t need to snapshot the client if we don’t in fact need it.

Now all that said, CRIU is an extreme example. Bring it up serves the purpose to show that solving these problems–if we make it a priority, and allocate the time to build a robust solution–is possible. We may not in fact need CRIU, but rather just to continue down the path that Velocity took. I’m not super informed about how Velocity paches code (and state). I quickly looked into their code to see how they were building their mirrors, but I think if the information could be shared with the rest of us, the community could devise a better solution.

@apendua @Sanjo can you guys tell us more about “injecting code at build level” and how we can improve that?

1 Like

That’s ok, I think Gagarin should focus on integration testing myself. Very soon Meteor 1.3 will have modules and then everyone can use the unit testing tools the JS community uses.

I’ve been using Karma & Jasmine for unit testing (via Webpack) and it works very well. Tests run in a few hundred ms or less.

End to End testing tools are very abundant and some like ChimpJS have support for talking to the server via DDP to reset the DB and do cleanup/setup work.

However, integration testing is very very badly needed in Meteor, which Gagarin fills nicely!


@SkinnyGeek1010 I agree with you. The only thing which is keeping us from using “normal tools” for unit testing is the lack of modules. Gagarin’s main feature is to manage a sandboxed meteor app environment, which shouldn’t even be needed if one’s doing real unit testing, not “integration-unit” testing.


Once Meteor 1.3 has modules, the biggest next steps in testing would be (IMO):

  1. A great integration test runner
  2. A Meteor testing toolbox for stubbing and mocking all parts of the Meteor core framework and recommended packages

So, just did some preliminary tests, and it looks like Gagarin and Nightwatch are going to work great together. I just added Gagarin integration to StarryNight, and have Nightwatch and Gagarin both using the same ChromeDriver installation. It’s doing just fine picking up tests from the package directories also.

starrynight run-tests --framework gagarin --verbose
starrynight run-tests --framework gagarin --webdriver http://localhost:9515
starrynight run-tests --framework gagarin --path /packages/*/tests/gagarin/*.js

We should eventually be able to either a) replace the Mocha instance that Nightwatch can access with Gagarin, so that Nightwatch has access to Gagarin’s integration functionality; or b) add a Nightwatch dependency to Gagarin to access the Nightwatch UI testing API in Gagarin. Either way, I’m pretty sure we should eventually be able to get something like the following syntax:

var nightwatch = require('nightwatch');

describe('Items List Page', function() {

  var client = nightwatch.initClient({
    silent : true

  var browser = client.api();

  before(function () {
    return client.execute(function () {
      Items.insert({_id: 'someFakeId', name: "Foo"});

  it("should be available on server", function () {
    return server.wait(1000, 'until data is propagated to the server', function () {
      return Items.findOne({_id: 'someFakeId'});
    }).then(function (value) {
  it('should display the item in a list', function (done) {
      .waitForElementVisible('body', 5000)
      .assert.title('Items List Page')
      .waitForElementVisible('body #itemsList', 1000)
      .assert.visible('body #itemsList .listItem:nth-child(1)')
      .assert.containsText('body #itemsList .listItem:nth-child(1) h2', 'Foo')


Also, since Gagarin shares the same Client/Editor/TestEnvironment isomorphism that Nightwatch is using, it's going to be pretty trivial to add Gagarin support to the StarryNight-Helper package for Atom.  Should have Atom integration finished by the end of the weekend.

Gagarin running in Atom (using the starrynight-helper package).

Atom’s package manager is a bit buggy, and my account is currently out of sync, waiting for a DB admin to fix things; so for the time being, anybody interested in tinkering with this will need to do the following to install:

git clone ~/.atom/packages/starrynight-helper

@awatson1978 Nice! I am really impressed :slight_smile:


Likewise! Gagarin has been a giant Christmas present for our team, and I’m tentatively going to say that we’ll be done with the FDA testing story once we get it fully integrated. Nightwatch Validation tests and Gagarin Verification tests. I think that may be a winner.

So, over the weekend, I set up a repository that includes all of the packages from the Clinical Meteor track, and got Gagarin to scan all of the packages in it! Only three of the packages currently have Verification test coverage, but initial work seems to indicate that this is going to work perfectly for package and distro maintainers.

The glob pattern is a really great approach, btw.