For those of you who have been looking to take your app to the next level, here’s how you get started with continuous integration with a Meteor app using Gagarin as the testing framework.
10 Likes
Thanks for writing this up and posting!
It’s interesting to see that Laika is still alive, and that it found it’s way to using Chai and Mocha also (so did SpaceJam, Nightwatch, and Velocity). And using the SpaceSuit marketing that SafetyHarness was trying to inspire… very, very fascinating.
For people looking for the original Gagarin documentation:
Wow, and the more I’m digging into this, it’s just amazing at how similar it’s become to Nightwatch’s integration with Mocha. Convergent evolution!
My initial impression is that we may be able to require Nightwatch at the beginning of a Gagarin test script, and get all of Nightwatch’s high-level UI testing API.
Thank you again for posting this. This is just fantastic work.
5 Likes
I posted this on the medium article but perhaps its a better question for discussion here:
Hey Sam, how are full stack integration tests looking (particularly in terms of publications)? Can you run tests that bootup an app with your publication available, subscribe to it from the client, then spy on and unit test calls made by the publication (say you got a publisher that calls lots of other functions to do advanced stuff), and then unit test the subscription results within various calls, eg. rendered React view components?
i.e. the holy grail of Meteor testing. How would we approach that setup with Gangrin?
@samcorcos @awatson1978 can you use tinytest for integration testing? I was under the impression you could but now I can’t seem to find anything on it.
Gagarin looks like the thing I was looking for to run the very few integration tests I have (i’m using chimp heavily and karma unit tests via webpack).
1 Like
Gagarin is superb. If I were maintaining Laika, it would be looks like this.
Gagarin has a fiber based API and here’s the guide on how to use it. We use this to write our UI tests for Kadira.
7 Likes
Holy smokes, Batman! The game has been changed. True integration testing for Subscriptions/Publications (highlight from @arunoda’s hackpad):
describe("simple gagarin test on the client", function() {
var app = meteor({flavor: "fiber"})
var client = ddp(app, {flavor: "fiber"});
it("should be passed", function() {
app.execute(function() {
// create posts collection and add a post
var Posts = new Mongo.Collection('posts');
Posts.insert({name: "firstOne"});
Meteor.publish('getPosts', function() {
return Posts.find();
});
Meteor.methods({
addPost: function() {
Posts.insert({name: Random.id()});
}
});
});
// subscribe to getPosts publication and wait for the ready message
client.subscribe('getPosts');
var posts = client.collection("posts");
expect(Object.keys(posts).length).to.equal(1);
// add a new post
client.call('addPost');
// wait until new data comes to the client
client.sleep(200);
// check the new data arrived or not
posts = client.collection("posts");
expect(Object.keys(posts).length).to.equal(2);
});
});
3 Likes
Depends on what you mean by an integration test. In principle, yes, TinyTest is sufficient… but in practice, if you have an actual matrix of integration tests, it requires creating a bank of reference apps and/or scripting the meteor tool and a CI server to create an autobuild system.
Thanks! That sounds like a pain 
I’m basically thinking of using it for testing models (mutation on client and server via meteor methods) and other things that might hit the database… most other things in my current systems can be pretty easily unit tested.
Sounds like Gagarin might be the ticket. I’m so tired of Velocity breaking every upgrade… though it’s state should be ‘stable’ for some time now 
2 Likes
I worked with Gagarin a while back, I enjoyed it. It was very intuitive to setup pre-conditions, which is sometimes the hardest part of testing. By ‘pre-conditions’ I mean something like : ‘user is logged in, user is on the home page, user has one friend request’.
Also it’s really fun to watch the tests happen, your entire application functionality you get to watch go by in a flash, if something goes wrong you can take a screenshot of the error. I’m looking forward to picking it back up again soon. @samcorcos great article.
1 Like
Hey Guys,
@faceyspacey thanks for calling me out! And thanks to @samcorcos for bringing Gagarin to everyone’s notice.
Gagarin has always been intended as an integration tests framework. It is a totally independent test runner which can be thought as an alternative for velocity, but only in the integration & end-to-end tests area. It does not provide any decent support for unit testing.
For those who have not used Gagarin yet, the approach to writing tests is very similar to what Laika had to offer. So basically the test runner is responsible for spawning meteor processes and connecting to a webdriver instance. Then, it sequentially executes routines on those entities by sending source code through either webdriver protocol or a DDP method.
The code snippet brought here by @faceyspacey is a good example of what we can achieve being able to run any code on the server side (like with meteor shell). There are still some sharp corners, e.g. loosing information about the function scope when it’s send to the server/client, that we will have to polish at some point, but apart from that everything works surprisingly well.
Our biggest problem at this point - as some of you have pointed out - is the lack of good documentation, which I never had time to finish. So any help in this area would be most welcome. Ideally, we should build a website contaning documentation along with some interesting examples.
When we have it, we can compare the current state of the art with whatever expectations the users may have. Actually, I am quite sure that a lot of features you may be thinking about are already implement - they’re just not documented properly.
If you have any questions related to using Gagarin, don’t hesitate and ask them here! For anyone interested, a very good source of examples is the Gagarin’s test suite itself:
Cheers,
Tomasz
6 Likes
Well, how can we add “decent support for unit testing.” Unit testing after all is the easiest form of testing, considering all the challenges we’ve had with integration testing and to a lesser extent end to end testing. If it’s just a matter of adding Unit Testing to Gagarin for it to be the Ultimate solution for testing in the Meteor world, then we should make it a top priority.
More generally, what problems does Velocity solve that Gagarin does not?
How can I help?
Hey,
I can help you to put some docs.
I did it here sometimes back: https://hackpad.com/Gagarin-Guide-RzdMvlwyYHV
(as I posted here)
May be shall we put this with the Meteor Guide.
@sashko any place for Gagarin in the Meteor Guide? May be in the future.
3 Likes
Yeah we’re still figuring out the testing story right now. We’re definitely taking a good look at Gagarin.
9 Likes
Hey Sashko, it would be great if Meteor developers were guided to utilize best practices. An out of the box model for writing and perform tests would raise the quality of Meteor developers, encourage their hire, and increase Meteor’s business adoption.
4 Likes
@arunoda I think it’s a great idea to use hackpad for scaffolding Gagarin guide. I think that creating a real documentation will require a much more structured work though.
@faceyspacey Let me state it this way: somewhere deep in my mind, there exists an ideal testing framework which integrates flawlessly with Meteor and is canonical and minimalistic in terms of the code you need to write to describe your testing scenarios. The problem is neither Velocity nor Gagarin is that “ideal” framework.
Velocity is really good in integrating your tests code with your application code (e.g. injecting code at build level) and it also provides a lot of freedom in terms of implementing custom frameworks on top of it. On the other hand Gagarin gives you much more control on your meteor test instances and simplifies communication between server and client because it’s run from outside the application itself. It’s not good for unit tests though because they tend to slow down dramatically.
By “they tend to slow down”, do you mean because of the startup time? I think we can reduce the startup time, if we would be able to connect to an already running instance of mongo, instead of booting up it’s own instance.
Besides that, one of the reasons we’ve implemented the glob pattern a while ago, is to be able to run only a selective set of tests. For example: gagarin packages/acl/tests/**.js -v to run only those in the “acl package”, or even specifying it further down.
Perhaps even further reduction in spawn time would be possible if gagarin had a watch flag, which spawns gagarin, but doesn’t end the process. Instead it should go watch the pattern for changes, and rerun those tests on change.
A big influence of gagarins startup time is the meteor build time though. And it seems to me that we cannot make that faster very easily. I’ve read about the performance gain of using webpack:webpack, (2 sec rebuildtime instead of the current 15+), but I haven’t tried it myself. However, if the ‘module patching’ they are advertising with really works, for both meteor and gagarin, then we can rerun all tests within a matter of seconds.
The ‘module patching’ that webpack is advertising with, combined with a filewatcher in gagarin, would be very powerfull.
1 Like
I think the information you hold when shared with the rest of us can help us all contribute to taking that ideal meteor testing framework from being “somewhere deep in [your] mind” to being a reality. Currently the information regarding making that ideal meteor testing framework and its challenges is only intimately known by a few (you, the guys behind Velocity, @arunoda, etc). I believe this dream needs to become a reality, and we need to make it a first class priority.
You’ve given a few abstract examples of the challenges faced here, but I think a conversation about the specifics (which already seems to be unfolding here) is in order if we ever plan to achieve this.
What does “injecting code at build level” actually mean (i.e. in depth)?
This seems to go hand in hand with “injecting code at build level.” It’s not efficient to build your application to run a unit test which can be more quickly run without being backed without your application–is that the idea? It takes gagarin on average like 30 seconds or something to build your app before your tests can run. Velocity has pulled off some tricks to do this faster by “injecting” code into an already built app? Is that the idea? It would be nice to do unit tests under the same hood as Gagarin, so you’re not using multiple tools for the same class of job.
It seems that the main issue isn’t even hot reloading of code, but rather patching state to the precise exact same “state” it needs to be to make your tests reproducible 100% of the time (along with any fixtures your tests may need). Cuz, we can hot reload code into the runtime pretty easily and basically in an instant for a small number of files changed (even with all the transpilation that goes on in build steps), but there is no guarantee that the application state will be exactly what it needs to be after running the first test, let alone after many tests. This is especially true given the opaque nature of closures, i.e. where state is hidden as private variables within closures. You could take snapshots of all of mini-mongo and put it back to where it was before each test, but it doesn’t mean other code won’t have changed private variables within closures. Would snapshotting and replacing the entire window object do the trick–I’m not sure, is there still artifacts lurking in the runtime?
On a side note, Redux through being unidirectional, functional, and immutable allows for such testing today (so long as you maintain its functional principles). For all apps to maintain these principles in all aspects of its code, it’s a hard pill to swallow, and definitely not possible for non-functional apps built already, and likely may never be fully possible even for new apps unless you wanna embrace the functional approach wholeheartedly. Even tools like React + Redux likely don’t get that write without being somewhat of a “leaky abstract”–something like Cycle JS would likely be a far better option (in pure js land). Even then, it comes at the loss of many of the good aspects of OOP, which–without making this the main topic for debate–is still the best tool for a great number of problems, and I would definitely say is an easier way to learn programming than functional programming (though, perhaps that’s like saying English is easier to learn than Chinese first because I learned English first). Regardless, OOP is definitely not going away and is still the predominant style of coding, so I think we should find a way to offer a lot of what Redux offers in terms of time traveling “debugging” today with traditional coding styles.
I’d like to point out what Tonic Dev ( https://tonicdev.com ) does to quickly run any Node JS code you run in the browser–read this blog post by them: http://blog.tonicdev.com/2015/09/10/time-traveling-in-node.js-notebooks.html. Basically, they use this somewhat new (or at least recently evolved) tool: http://criu.org/Main_Page to snapshot the entire NodeJS runtime, as well as its associated file system, at a lower level, similar to how VM containers are made. They can then roll back the entire application state flawlessly.
I’ve been looking into such things because I refuse to believe that the state of Velocity or the build times of gagarin will hold us back from the true benefits of TDD: basically immediate response and observability of your code. In short, with the current limitations the only benefits we are getting from tests is protection from regressions. While that’s important, I think the upgrade in development flow of true immediate TDD is in a world of its own. Check out Wallaby JS right now if you haven’t:
here’s the video tour:
As you can see, you basically have what Chris Granger’s Light Table was only effectively doing with ClojureScript (their glorified JS support never reached much more than a basic JS repl last time I checked). So Wallaby is doing similar stuff where it shows the results of your tests and inline errors within your tests and within the tested code AS YOU TYPE. We could only dream of such things without first getting our build time in order, or rather, workarounds where only changed code can both be patched, while syncing application state back to a previous snapshot.
Being able to code this way would get the full power out of tests. It would lead to a lot more tests, because more often people would start with them, rather than end with them just to catch regressions. In short, without such things TDD isn’t even a productive workflow for Meteor development today. It’s extra work that clients often don’t pay you to do, whose benefits aren’t immediately apparent.
Where my train of thought goes next is whether CRIU could snapshot the browser runtime (and replace it), not just Node JS on the server?? Because even if we were able to implement CRIU for server side testing, the client app still needs to be setup and torn down equally as quickly.
Now that, said, because of React, having to test your code on the client is becoming less and less important. The following project https://github.com/bruderstein/unexpected-react allows you to use JSDOM to render React on the server and test it. In addition, you don’t even need to JSDOM if you want to test React components individually. You can use React’s built-in testing addon. The problem with that though is that it only shallowly tests the output of your React render methods, i.e. for its precise return, not the return of nested child components. You need a DOM to do do that. What that means is if you change how you compose your components, but what’s written to the DOM is still the same, the test based on React’s testing utilities will fail, whereas the DOM-based tests won’t. So that’s where JSDOM essentially puts ReactTestUtils to shame. …So anyway, perhaps we don’t need to snapshot the client if we don’t in fact need it.
Now all that said, CRIU is an extreme example. Bring it up serves the purpose to show that solving these problems–if we make it a priority, and allocate the time to build a robust solution–is possible. We may not in fact need CRIU, but rather just to continue down the path that Velocity took. I’m not super informed about how Velocity paches code (and state). I quickly looked into their code to see how they were building their mirrors, but I think if the information could be shared with the rest of us, the community could devise a better solution.
@apendua @Sanjo can you guys tell us more about “injecting code at build level” and how we can improve that?
1 Like
That’s ok, I think Gagarin should focus on integration testing myself. Very soon Meteor 1.3 will have modules and then everyone can use the unit testing tools the JS community uses.
I’ve been using Karma & Jasmine for unit testing (via Webpack) and it works very well. Tests run in a few hundred ms or less.
End to End testing tools are very abundant and some like ChimpJS have support for talking to the server via DDP to reset the DB and do cleanup/setup work.
However, integration testing is very very badly needed in Meteor, which Gagarin fills nicely!
2 Likes