Velocity Update: Xolv.io are handing Meteor testing back to the MDG


#1

In July 2013, the MDG announced “Velocity: the official testing framework for Meteor applications”.

In July 2014, the MDG announced their plans to “to move Velocity into core and simplify and streamline it.”

TL;DR: Xol.vio are no longer supporting Velocity and are handing back the responsibility of the “Official Testing Framework” to the MDG.

We are proud to have given birth to Velocity and now that it has paved the way for Meteor testing and has become more mature, it needs dedicated support at the core.

We will continue to push the boundaries of testing through our own endeavours, and we are currently focused on two main products:

Chimp: Our open-source testing library that makes it super easy to automate end-to-end tests. Chimp is both an npm and a Meteor package and works with or without Velocity.

Read more about Chimp →

Simian: Our commercial product that simplifies the gathering of product requirements through team-wide collaboration.

Read more about Simian →

What you need to know

  • Will Xolv.io Continue to Maintain Their Meteor Testing Packages?

    Yes. We are a company that helps other companies innovate faster through testing and automation. We practice what we preach and we need to use a testing framework that allows us to test our own products. Currently the best way that we know and recommend to do this in effectively in Meteor is to use sanjo:jasmine and xolvio:cucumber. We will continue to maintain these packages for our clients and for our needs, and we will continue to share our work with the community.

    Our testing frameworks require test environments to run against when running in development mode (more on this below). Currently Velocity does this using mirrors therefore the Velocity core will still power our test packages until a better option becomes available. We’ll switch our frameworks to use the best means as they become available.

  • So what actually happens to Velocity?

    1. The Velocity website will be replaced with a list of testing options for Meteor.

    2. The Velocity GitHub issues will be closed and frameworks that continue to use the Velocity core will handle their own support requests.

    3. We expect the MDG will clarify what “The Official Testing Framework for Meteor” will be in the future.

  • Will Xolv.io Continue to offer Premium Meteor Testing Support?

    Yes. We will continue to offer our expertise, training and Meteor testing support as part of our company services. Please see our Meteor Testing Support for more details.

  • What about The Meteor Testing Manual?

    The book will continue to be written and to evolve as the Meteor testing landscape changes, it is The Meteor Testing Manual after all! The book is about how to test Meteor apps and is not married to Velocity. If anything, we will now have more time to write it.

  • When is all this happening?

    It just happened. Over the next few weeks, we will update all the pages, blurb, websites and such. We are proud of our role within the Meteor testing story and look forward to hearing how the MDG will take it forward for the future.

Velocity Retrospective

History

Velocity was born in April 2014 as a result of a meeting between the major testing framework authors, each of which had their own role in the Meteor testing story. When we all came together, we proposed the following goals for a unified testing framework:

  1. Simple installation
  2. Flexible for the community to evolve
  3. An easy Meteor-like workflow
  4. A one-stop-shop for all Meteor testing

In July of the same year, it was endorsed by the MDG as the “Official Testing Framework” for Meteor. Fueled by the challenge and responsibility of this undertaking, we set out to create the Meteor testing solution to rule them all. The team included Mike Risse (mike:mocha) and Jonas Aschenbrenner (sanjo:jasmine). Let look at each goal above:

1. Simple installation

It’s really easy to install Velocity-based frameworks. You type meteor add sanjo:jasmine or meteor add xolvio:cucumber and you’re good to go. Some frameworks chose not to follow this ethos, but most did and there have been little to no complaints in this area.

Success.

2. Flexible for the community to evolve

Velocity was built from the ground up as a set of APIs that framework authors can use to build any testing framework or plugin. There are 6 testing frameworks on the Velocity homepage, and we are still seeing people extend and use Velocity in their own applications and packages.

Success.

3. An easy Meteor-like workflow

Velocity-based frameworks integrate nicely into the Meteor workflow. You can reactively see test results inside your app as you develop. The experience is far from perfect and can certainly do with the simplifying and streamlining love the MDG announced. More on this below in the test environments section.

Success… needs love.

4. A one-stop-shop for all Meteor testing

This was arguably the reason that the Velocity team got together; to unify our efforts as test-framework authors and combine our brains and smarts to create a one-stop-shop for Meteor developers. Today we still have a lot of confusion when it comes to testing with Meteor.

Learning: The MDG needs to express an opinion on how testing should be done with Meteor.

Velocity’s Triumphs

In figuring out how to test Meteor applications, we needed to figure out how test all 7 testing modes of Meteor (see this blog post). This journey led to the following outcomes:

Package Testing using Jasmine / Mocha

Mike was doing some work for the Respond.ly team and figured out the pathway to using Mocha to run Meteor package tests. Soon after, Jonas generalized the solution and now you can use both of these fully-features industry standard frameworks to write package tests, instead of the primitive TinyTest.

App-level Client / Server Integration Testing

Both Jasmine and Mocha allow you to run tests within your running app’s Meteor context. Whilst the package testing approach provided by TinyTest / SpaceJam / Mocha / Jasmine above provides you this in a package, it forces you to split your application up into packages in order for your application to be testable. This is not a reasonable ask as the package-only architecture is just one way to build apps.

App-level Client / Server Unit Testing

Sanjo went further with Jasmine and supported unit-testing modes. These tests run at hyper-speed and run without the Meteor context (they still need the Meteor to be running to establish a context. See more about this in the challenges section below).

App-level End-to-End testing

The nature of end-to-end testing frameworks is to run outside the context of the system under test, however running end-to-end tests reactively within the developer workflow gave rise to the @focus tag in xolvio:cucumber, which makes outside-in testing a lot smoother. Also, end-to-end applications need fixtures to setup data, and from this need the fixture-package pattern was created for sharing test-data setup code between integration testing frameworks and end-to-end frameworks.

Changes to the Meteor Core

In Q4 of 2014, the MDG added the debugOnly only flag as a result of a meeting between us all, this flag is used to keep applications out of production builds. This has enabld other development-only tools to be possible, such as the infamous Meteor Toys.

Not too long after that, Meteor supported the --test command to run Velocity-based framework tests. This runner can be seen here.

Incredible Uptake

At the time of writing this post, there are nearly 320k downloads of Velocity-based testing frameworks on Atmosphere.

Velocity’s Challenges

To make the above happen, a lot magic was needed. Here are the issues that we have not been able to resolve:

Test Environments (AKA Mirrors)

This has been one of the toughest challenges for us.

When developing a Meteor application, the developer workflow is to start Meteor and start coding. To run tests reactively at the same time as building an app, we need to make sure the tests do not overwrite the main application data. For this to happen, the tests run inside a mirror, which is another word for a test environment.

In fact, this is how the TinyTest package testing mode works, it loads the packages and test files that you specify in the onTest section and provides you with an integration context, separate to your main application. Mirrors do the same, but they include the entire application context.

Our first approach to mirrors was called an rsync-mirror. This mirror used rsync to create a replica of the main app on the filesystem, and a second Meteor process was started on another port using the copied filesystem. This was great for doing things like instrumenting files and adding custom files/packages for testing, but the experience was not responsive enough, especially when the user would save files quickly (amongst a ton of other issues). So we scrapped it.

Our second approach was called a soft-mirror. It would wait for the Meteor build system to complete and then kick off a node process on the main.js file. This version would start within 30ms after Meteor started. We thought we hit jackpot! Unfortunately, it was not easy to load additional files that were needed for testing inside the test environment. So we created a test-proxy package. This package would scan the filesystem and add test files to the mirror only. Whilst the speed was amazing, the test-proxy approach was ridden with start/restart bugs, so we scrapped it.

Our third (and current) approach was to fork Meteor itself and modify it to make it support testing. It allows us to specify additional source locations and build file output directories. This approach has proven to be the most stable but it’s still not perfect. The startup time of Meteor has been getting slower of late and with every release, we have to update a new Meteor track.

The first and third approach are both flawed in their need to run multiple build processes. This is where most of the CPU cycles get eaten up, along with the memory requirements becomin huge. When you consider that a lot of people use 3 mirrors, that’s 4 Meteor apps running in parallel - a lot even for a mighty machine.

The reason Velocity is slow is due to mirrors.

What we really need, is the best of all the above approaches. A test environment where we can:

  • Load the test files we want into it
  • Specify the packages we want to include / exclude
  • Starts in milliseconds and in tandem with the Meteor lifecycle
  • The ability to instrument files (not essential but highly desirable for test coverage reports)

Deep Coupling

Unit testing in Meteor is non-trivial. The framework did not take into account unit testing from the outset and as such, it’s hard to isolate units of code without fully loading the Meteor context, or a stubbed version of it. Perhaps when Meteor supports modules it will become easier to do this, but right now it’s non-trivial.

This decoupling is what is needed for super-fast testing. Luckily, it is possible to achieve the same speed from integration tests using test doubles (like spies), but you still need to first start Meteor in order to get the context.

It would be amazing if the MDG did released an official Meteor stub with every Meteor release. The packages that your app needs may also need to be stubbed, but that’s not nearly as difficult as most packages tend to export a single namespace and it’s fairly easy to create test doubles.

Closing Thoughts

We have really enjoyed carving out the Meteor testing landscape and are proud of all the triumphs and to have persisted through the challenges.

We send the MDG our best wishes and look forward to seeing how they march forward with Meteor testing from here.

With best regards, your friends at Xolv.io

(:


The State of Velocity and Testing in Meteor
#2

How do I get to be the first commenter ?

Sam, Thank you so much for your efforts to move testing forward in Meteor. You and everyone mentioned above have made many newbies aware of testing.

It should concern us all that we remain with the original problem of providing (fast) testing unsolved, and you point out some of the ways (mirrors are complex and slow) in which those who want/expect/require fast tests may be disappointed for some time to come. I’m eager to take a look at a new, module based Meteor, but I wouldn’t bet on testing being ‘solved’ quite as quickly as that.

But all in all this is an awesome post, capping off an important era for Meteor - and now that I’m working deep on a module-based Node project, I’ll be happy to show simian/chimp around my to my current clients.


#3

I just wanted to say thanks to the Velocity team and say how much I appreciate all of your efforts!


#4

One thing that’s a plus… if you’re using ES6 modules (webpack for now, core later) the outlook is really nice!

You can use standard JavaScript testing tools to unit test your code. It’s super fast too because you don’t go through the build step… i’m getting sub 1 second pass/fail specs with Karma + Jasmine in Chrome.

Wrapping your data layer with something like a repository pattern makes things much easier to unit test. Most Meteor bits that i’ve ran into can be stubbed with a simple Jasmine stub (opting for exportable functions instead of anonymous funcs helps too).

End to end tests don’t require velocity either so those don’t have to go through Velocity either. I used to use Rspec + Capybara for this but i’m looking into Chimp now as an alternative for cucumber.

The remaining bits could be tested with integration tests via Velocity but IMHO if you have high coverage with unit and end-to-end than your overall confidence should be high. Even if you have a few integration tests for publications you can still run Velocity in CI mode as a sort of smoke test instead of watching continuously.


#5

Good to see a decision has been made @sam!


#6

Thanks a lot for all of your and @Sanjo’s work on this!


#7

@sam is there a date (timeframe) you can share for the release of the testing manual?

I remember pre-paying for a testing book about (probably more than) a year ago and receiving an email about that book getting folded into the testing manual. But no more news ever since :disappointed:


#8

Thank you Xolv.io for all that you have done for meteor testing.

Now, i’m a litlle worried by this announcement.
In July 2015, MDG announced their plans to "to move Velocity into core and simplify and streamline it."
Today Xolv.io is announcing that they are “handing back the responsibility of the “Official Testing Framework” to the MDG.”

I may be wrong on this, but for me, since this is a Xolv.io only announcement, it means that Xolv.io is tired of waiting for MDG to begin work on testing in Meteor.

In mars of 2014, before 1.0 which was to be released without testing support, i said that MDG seemed to have no real interest in testing. Someone from MDG responded that they were “commited” to testing and that a more relevant question would be “when” they would get to this topic. One year and a half later, i still stand by my comment.


#9

You should have access to it online if you purchased the Testing Meteor book I was working on. If you didn’t get that access, feel free to contact @sam directly to handle that.


#10

And I am sure they still stand by their comment too. I think this is a time/people issue more than anything. The real tragedy here is that you had a group of community people interested in doing something and MDG did a poor job of helping them achieve this goal…


#11

Oh! Great! Thank you for this information.


#12

@deanius @SkinnyGeek1010 @rjakobsson @vjau thank you all for the kind words

Couldn’t agree more with both of the statements. Wrapping and introducing indirection is the best approach for decoupling for any framework, with the caveat that the indirection could be misconfigured and the unit / end-to-end tests would still pass.

Way I see it, E2E tests ensure the value to the user is being met. Integration tests make sure you wired up and configured your components together. Unit tests validate the logic.


#13

Thanks for your work, and I’m anxiously eager to hear from MDG about next steps.


#14

Hey Sam,

First off, thanks so much for leading the charge in the community on testing and Meteor. I think you and the others in the Velocity team have done a great job of picking up the considerable slack.

Also, thanks to you and Jonas for helping and advising Sashko and myself with the guide and especially with the testing article. I’m still trying to get a complete picture of everything that’s out there, and it’s been very helpful to hear from you guys, as you are clearly at the forefront of the space.

Something that we think will come out of the guide is a good picture of what the landscape looks like today (with testing and a lot of other areas of Meteor). Of course with the advent of ES2015 modules, and the potential for webpack integration, that landscape is in flux for testing in particular. But I hope and expect this will lead to some obvious low-hanging fruit that can be picked off both by the community and by us at MDG.

With that in mind, it’s great to hear these war stories about Velocity, and I’d love to hear more about some of the lessons learned, and your thoughts on the best ways forward. For instance, I’d love your comments on this ticket.


#15

I think the lack of consideration for testing and modularity in the current version of Meteor has been a huge problem. Any serious software engineering requires these things so that a team of people can work together on a project without worrying that each change might break something. Refactoring becomes nearly impossible without the safety harness of good tests.

In my mind it’s a side effect of the way Meteor’s development has historically emphasized a low-boilerplate and easy to get started with design, but at huge expense in terms of encouraging good software engineering practices.

I believe the Meteor Guide is a big step in the right direction - the intent is to describe all of the things one would need to do to build a “serious” Meteor app, and I believe that it can unshackle Meteor from the world of 1-hour long getting started tutorials and remind people to think about app organization, maintenance, and reliability. Perhaps the first version won’t have a perfect testing story, but at least there will be some obvious gaps that could drive development of a fix.

Another great step will be the introduction of a modern JavaScript module system in Meteor 1.3. Hopefully soon after we can rewrite large parts of Meteor to be less global and magical, so that you can stub out modules or some of their imports or exports. I hope this will also enable the Meteor community to use more off-the-shelf JavaScript testing tools that know more about how to deal with these modules - for example, Jest or Istanbul.


#16

More thanks to @erichsu and @tmeasday for the kind words.

I will surely comment. My intention with this thread was to provide the feedback in a retrospective format so that the MDG can run with. We should have a hangout to give you a good download of the lessons learnt - I think that will help you a lot with creating the testing app story. We have a lot of demos and use-cases we can share.

[quote=“sashko, post:15, topic:13117”]
Another great step will be the introduction of a modern JavaScript module system in Meteor 1.3. Hopefully soon after we can rewrite large parts of Meteor to be less global and magical, so that you can stub out modules or some of their imports or exports. [/quote]

Great to hear about 1.3 bringing modularity, we’ve definitely been waiting for it and it will be an enabler for unit testing. All positive things.

I just want to highlight that supporting test environments is still necessary. Not only in the interim to cover the case of testing of not-yet-decoupled code, but also for integration testing in general as noted in my comment above:

Currently the only way to do integration testing is to use a Velocity compatible integration testing framework, and this will continue to be the case after the modularity.

In the Java world for example, I can run a build that only includes my integration-test using Maven profiles. The same is possible in many other build systems like msbuild, ant, nant, you name it!

For E2E testing, test environments are not essential but they are still highly desirable. I would want to include packages that provide me back-door access in an e2e testing environment but not to production. Interestingly, it’s easy to do that in TinyTest / Velocity package testing using the onTest api.

What I’m really saying here is that the build system is another accomplice to the difficulties in testing. The current Velocity tracked release of Meteor fixes this by allowing mirrors to compile code inside the tests directory for a given framework, and by outputting the compiled files to another directory. Merging this into the core is probably the quickest win to getting integration testing support, but I think the long term solution should be to upgrade the build system, or to use an industry standard one.

Anyway, excited to see the modularity refactor. If this includes a revamp of the build system, then hallelujah and in both case, I will soon be happier than now :smile:


#17

This sounds really promising. Appreciate all your share with the community. Would love to see a tutorial on this if you find the time :smile:


#18

Hey Sam,

First of all thanks for all your great efforts! I just started to learn velocity and jasmine, so I’m not sure where should I ask my questions. I already wrote bunch of unit and integration tests and now, when velocity docs are shut down - what should I do next? Should I start looking at some another framework? If currently I should stick with velocity - is there any chance to get all docs and examples from https://velocity.readme.io , cause they were very helpful.

And btw, is there any ETA when MDG will start support their own testing framework?


#19

We continue to maintain sanjo:jasmine and xolvio:cucumber, because we use it. The documentation is still available under new URLs. The new locations are linked here: https://velocity.readme.io/.


#20

thanks @tsepelev

I can’t tell you what the MDG will support and when, but my advice is to continue writing tests using Jasmine until the MDG/community come up with a better solution.

As discussed above, the main gap of Meteor testing that Velocity plugs, is the ability to do app-level unit / integration testing, and as Sanjo said, we will continue to use our own Velocity-based testing frameworks in the meantime and share with the community. Just be mindful that it’s not the “official testing framework”.

The good thing is that there are two main testing tools in the JS community: Mocha and Jasmine. I can only imagine one of those will be picked in the long run, and to convert your tests from Jasmine to Mocha is very easy, it’s just a little syntax.

On the plus side, app-level end-to-end testing is completely within our control since it does not require any changes to the Meteor core. We are focusing on this with our Chimp npm package that already works without Velocity. You can run it against either your main app, or start your own parallel app (mirror) from the command line, then start Chimp this way:

chimp --watch --ddp=http://localhost:3000

Chimp downloads Selenium for you, initializes a webdriver connection, initializes a DDP connection, fiberizes everything so you have no callbacks, no promises, adds a watch mode to any features you tag with @watch. All this means you can just write tests in your favorite testing language. Currently it works with Mocha & Cucumber and we’ll be adding Jasmine. I’ll make a fuller announcement + tutorial soon.


Meteor Application Testing: a minimum viable design