In July 2013, the MDG announced “Velocity: the official testing framework for Meteor applications”.
In July 2014, the MDG announced their plans to “to move Velocity into core and simplify and streamline it.”
TL;DR: Xol.vio are no longer supporting Velocity and are handing back the responsibility of the “Official Testing Framework” to the MDG.
We are proud to have given birth to Velocity and now that it has paved the way for Meteor testing and has become more mature, it needs dedicated support at the core.
We will continue to push the boundaries of testing through our own endeavours, and we are currently focused on two main products:
Chimp: Our open-source testing library that makes it super easy to automate end-to-end tests. Chimp is both an npm and a Meteor package and works with or without Velocity.
Simian: Our commercial product that simplifies the gathering of product requirements through team-wide collaboration.
What you need to know
-
Will Xolv.io Continue to Maintain Their Meteor Testing Packages?
Yes. We are a company that helps other companies innovate faster through testing and automation. We practice what we preach and we need to use a testing framework that allows us to test our own products. Currently the best way that we know and recommend to do this in effectively in Meteor is to use
sanjo:jasmineandxolvio:cucumber. We will continue to maintain these packages for our clients and for our needs, and we will continue to share our work with the community.Our testing frameworks require test environments to run against when running in development mode (more on this below). Currently Velocity does this using mirrors therefore the Velocity core will still power our test packages until a better option becomes available. We’ll switch our frameworks to use the best means as they become available.
-
So what actually happens to Velocity?
-
The Velocity website will be replaced with a list of testing options for Meteor.
-
The Velocity GitHub issues will be closed and frameworks that continue to use the Velocity core will handle their own support requests.
-
We expect the MDG will clarify what “The Official Testing Framework for Meteor” will be in the future.
-
-
Will Xolv.io Continue to offer Premium Meteor Testing Support?
Yes. We will continue to offer our expertise, training and Meteor testing support as part of our company services. Please see our Meteor Testing Support for more details.
-
What about The Meteor Testing Manual?
The book will continue to be written and to evolve as the Meteor testing landscape changes, it is The Meteor Testing Manual after all! The book is about how to test Meteor apps and is not married to Velocity. If anything, we will now have more time to write it.
-
When is all this happening?
It just happened. Over the next few weeks, we will update all the pages, blurb, websites and such. We are proud of our role within the Meteor testing story and look forward to hearing how the MDG will take it forward for the future.
Velocity Retrospective
History
Velocity was born in April 2014 as a result of a meeting between the major testing framework authors, each of which had their own role in the Meteor testing story. When we all came together, we proposed the following goals for a unified testing framework:
- Simple installation
- Flexible for the community to evolve
- An easy Meteor-like workflow
- A one-stop-shop for all Meteor testing
In July of the same year, it was endorsed by the MDG as the “Official Testing Framework” for Meteor. Fueled by the challenge and responsibility of this undertaking, we set out to create the Meteor testing solution to rule them all. The team included Mike Risse (mike:mocha) and Jonas Aschenbrenner (sanjo:jasmine). Let look at each goal above:
1. Simple installation
It’s really easy to install Velocity-based frameworks. You type meteor add sanjo:jasmine or meteor add xolvio:cucumber and you’re good to go. Some frameworks chose not to follow this ethos, but most did and there have been little to no complaints in this area.
Success.
2. Flexible for the community to evolve
Velocity was built from the ground up as a set of APIs that framework authors can use to build any testing framework or plugin. There are 6 testing frameworks on the Velocity homepage, and we are still seeing people extend and use Velocity in their own applications and packages.
Success.
3. An easy Meteor-like workflow
Velocity-based frameworks integrate nicely into the Meteor workflow. You can reactively see test results inside your app as you develop. The experience is far from perfect and can certainly do with the simplifying and streamlining love the MDG announced. More on this below in the test environments section.
Success… needs love.
4. A one-stop-shop for all Meteor testing
This was arguably the reason that the Velocity team got together; to unify our efforts as test-framework authors and combine our brains and smarts to create a one-stop-shop for Meteor developers. Today we still have a lot of confusion when it comes to testing with Meteor.
Learning: The MDG needs to express an opinion on how testing should be done with Meteor.
Velocity’s Triumphs
In figuring out how to test Meteor applications, we needed to figure out how test all 7 testing modes of Meteor (see this blog post). This journey led to the following outcomes:
Package Testing using Jasmine / Mocha
Mike was doing some work for the Respond.ly team and figured out the pathway to using Mocha to run Meteor package tests. Soon after, Jonas generalized the solution and now you can use both of these fully-features industry standard frameworks to write package tests, instead of the primitive TinyTest.
App-level Client / Server Integration Testing
Both Jasmine and Mocha allow you to run tests within your running app’s Meteor context. Whilst the package testing approach provided by TinyTest / SpaceJam / Mocha / Jasmine above provides you this in a package, it forces you to split your application up into packages in order for your application to be testable. This is not a reasonable ask as the package-only architecture is just one way to build apps.
App-level Client / Server Unit Testing
Sanjo went further with Jasmine and supported unit-testing modes. These tests run at hyper-speed and run without the Meteor context (they still need the Meteor to be running to establish a context. See more about this in the challenges section below).
App-level End-to-End testing
The nature of end-to-end testing frameworks is to run outside the context of the system under test, however running end-to-end tests reactively within the developer workflow gave rise to the @focus tag in xolvio:cucumber, which makes outside-in testing a lot smoother. Also, end-to-end applications need fixtures to setup data, and from this need the fixture-package pattern was created for sharing test-data setup code between integration testing frameworks and end-to-end frameworks.
Changes to the Meteor Core
In Q4 of 2014, the MDG added the debugOnly only flag as a result of a meeting between us all, this flag is used to keep applications out of production builds. This has enabld other development-only tools to be possible, such as the infamous Meteor Toys.
Not too long after that, Meteor supported the --test command to run Velocity-based framework tests. This runner can be seen here.
Incredible Uptake
At the time of writing this post, there are nearly 320k downloads of Velocity-based testing frameworks on Atmosphere.
Velocity’s Challenges
To make the above happen, a lot magic was needed. Here are the issues that we have not been able to resolve:
Test Environments (AKA Mirrors)
This has been one of the toughest challenges for us.
When developing a Meteor application, the developer workflow is to start Meteor and start coding. To run tests reactively at the same time as building an app, we need to make sure the tests do not overwrite the main application data. For this to happen, the tests run inside a mirror, which is another word for a test environment.
In fact, this is how the TinyTest package testing mode works, it loads the packages and test files that you specify in the onTest section and provides you with an integration context, separate to your main application. Mirrors do the same, but they include the entire application context.
Our first approach to mirrors was called an rsync-mirror. This mirror used rsync to create a replica of the main app on the filesystem, and a second Meteor process was started on another port using the copied filesystem. This was great for doing things like instrumenting files and adding custom files/packages for testing, but the experience was not responsive enough, especially when the user would save files quickly (amongst a ton of other issues). So we scrapped it.
Our second approach was called a soft-mirror. It would wait for the Meteor build system to complete and then kick off a node process on the main.js file. This version would start within 30ms after Meteor started. We thought we hit jackpot! Unfortunately, it was not easy to load additional files that were needed for testing inside the test environment. So we created a test-proxy package. This package would scan the filesystem and add test files to the mirror only. Whilst the speed was amazing, the test-proxy approach was ridden with start/restart bugs, so we scrapped it.
Our third (and current) approach was to fork Meteor itself and modify it to make it support testing. It allows us to specify additional source locations and build file output directories. This approach has proven to be the most stable but it’s still not perfect. The startup time of Meteor has been getting slower of late and with every release, we have to update a new Meteor track.
The first and third approach are both flawed in their need to run multiple build processes. This is where most of the CPU cycles get eaten up, along with the memory requirements becomin huge. When you consider that a lot of people use 3 mirrors, that’s 4 Meteor apps running in parallel - a lot even for a mighty machine.
The reason Velocity is slow is due to mirrors.
What we really need, is the best of all the above approaches. A test environment where we can:
- Load the test files we want into it
- Specify the packages we want to include / exclude
- Starts in milliseconds and in tandem with the Meteor lifecycle
- The ability to instrument files (not essential but highly desirable for test coverage reports)
Deep Coupling
Unit testing in Meteor is non-trivial. The framework did not take into account unit testing from the outset and as such, it’s hard to isolate units of code without fully loading the Meteor context, or a stubbed version of it. Perhaps when Meteor supports modules it will become easier to do this, but right now it’s non-trivial.
This decoupling is what is needed for super-fast testing. Luckily, it is possible to achieve the same speed from integration tests using test doubles (like spies), but you still need to first start Meteor in order to get the context.
It would be amazing if the MDG did released an official Meteor stub with every Meteor release. The packages that your app needs may also need to be stubbed, but that’s not nearly as difficult as most packages tend to export a single namespace and it’s fairly easy to create test doubles.
Closing Thoughts
We have really enjoyed carving out the Meteor testing landscape and are proud of all the triumphs and to have persisted through the challenges.
We send the MDG our best wishes and look forward to seeing how they march forward with Meteor testing from here.
With best regards, your friends at Xolv.io
(:
