Cucumber v0.8.0 - The Most Comprehensive End-to-End Testing Framework for Meteor

Announcing Cucumber v0.8.0

meteor add xolvio:cucumber

This updated has taken over 2 months of work to complete and is built from the ground up on the lessons learnt from working with clients, working with the community, writing The Meteor Testing Manual and working on our own startup.

It’s built on top of the rock-solid Velocity 0.6 and sports the following features:

Fluent Syntax
You now have a fluent syntax using promises that allows you to write concise and clean readable code inside your steps like this:

return this.browser.
   click.('.this-class').
   getText('.that-class').should.become(title)

Notice there are no callbacks! This is because I’ve baked-in WebdriverIO’s promises with Chai-as-promised into steps as well as promisifying CucumberJS. You can still use callbacks if you need to of course.

Streamlined Meteor-Style Development Mode
Many said it doesn’t make sense to run end-to-end tests on save whilst developing like everything else in Meteor. Well I beg to differ! This update will only run scenarios that you tag with @dev in your feature files. This means you can focus on building your app one scenario at a time, and on the CI server all the scenarios will run (unless you @ignore them of course).

Pre-Wired DDP Connection
In the step definitions, you now have access to a ddp connection that allows you to call methods on the mirror. This is invaluable in allowing you to setup test data with simple calls like:

this.mirror.call('setup-users', 'tom', 'jim')
// or
this.mirror.apply('setup-users', ['tom', 'jim'])

Real Browsers with Session Reuse
You can now choose any browser by running SELENIUM_BROWSER=chrome meteor. This will use chrome instead of the default phantomjs. You can of course also use firefox or safari and even iexplorer. When you usually use a real browser, you get an annoying flickering of windows appearing and resizing. Not with this package! Your browser window will stay open and will get reused throughout your development session. And don’t worry about not having Selenium, this package will auto download everything for you.

SauceLabs / BrowserStack / Selenium Grid Support
Want to test on a variety of browsers? No problem! You just enter your host address along with a few other details and you’re all set. See here for instructions.

Slick console reporting
The latest Velocity logging update means you can now tail the cucumber logs separately to other frameworks and the main app. This means as your steps run, you can keep your eye on the console and get detailed feedback. This package now cleans up the logs by allowing you to copy/paste auto-generated step snippets without timestamps and you even get fancy colors for clarity.

Screenshots on Errors
When a test fails, you need all the info you can get. So in addition to the detailed console reporting, you also get a screenshot when something goes wrong. This is especially useful when you’re using chrome, or running on CI. If you don’t like the default output directory, you can customize it.

Not Just Meteor
Whilst xolvio:cucumber is a meteor package, it uses cuke-monkey under the covers, which is built for use in any platform where users wish to use cucumber. If you’re interested in, check out the Github repository.

Breaking changes
All of the above came at a little price of some breaking changes. If you are on a version < 0.6.0, please be sure to follow these migration instructions.

All of the above are now out-of-the-box, just upgrade and let us know what you think. Please report any issues that you find on the GitHub issue page.

Next
The super fast parallel testing work is being done as you read this so expect that update soon. If you don’t know what htis is, you should check out the devshop presentation Mike Risse and I did about it here.

13 Likes

sounds very cool, can’t wait to try it out.

so the github page is the best place for getting started ?

also not being a cucumber expert, but can this be used for simple server side unit tests, or is it mostly for driving the web browser acceptance tests?

Is there a simple way to change the base URL the tests run against, eg if we want to run a test suite against our production site after deployment?

you also get a screenshot when something goes wrong

even when tests don’t fail, is there an easy way to take screenshots? where do these get saved to, esp if running in a CI location?

one typo in your code above, you have an extra dot.

differnet from what’s on your site:

this.ddp.call('yourMethod', [], callback);

Sam,

Thank you for all the effort you have put forth into this. I listened in to the Cucumber section on the MC Podcast and this post refreshed it in my mind! Going to give this darn testing thing a shot!

Sorry it took so long for me to respond, I’ve been battling with some Windows bugs and they are fixed in v0.8.0, along with some other small tweaks.

Instead of using this.ddp, the syntax is now this.mirror.call() / this.mirror.apply(). We actually patched the node ddp client to be promise enabled AND to follow the same syntax as meteor.

@dcsan I’ve updated the text, thanks for the headsup. The GitHub page is the best place to get started, please let me know if I need to update something on it. I’ve so many places to keep up to date now!

Short answer, the only tests are end to end currently using either the browser or DDP connection.

Long answer, currently the feature code is not linked to the main app at all which means you can’t do unit / integration testing with cucumber… yet! I can’t promise anything, but Sanjo and I are talking about how we can make this possible so that cucumber will allow you to inject code into the mirror at run-time and perform integration tests. A similar approach can be done with unit tests.

Yes, you can use cuke-monkey instead of using meteor-cucumber and run it against any environment. In fact, meteor-cucumber is just a wrapper around cuke-monkey which handles all the complexity. I still need to document the cuke-monkey project so feel free to ask how to configure that on the issue there and I’ll guide you.

Absolutely, see the webdriver docs here

@hoytevans let me know how you fare.

For anyone that is having trouble with the latest release, please delete the .meteor/local directory first and then try again. If you have issues, post them to:

Ensuring you do this

  • run with VELOCITY_DEBUG=1 && meteor
  • paste logs from both main process and mirror in issue
  • paste .versions file in issue

Thanks

Short answer, the only tests are end to end currently using either the browser or DDP connection.

Long answer, currently the feature code is not linked to the main app at all which means you can’t do unit / integration testing with cucumber… yet! I can’t promise anything, but Sanjo and I are talking about how we can make this possible so that cucumber will allow you to inject code into the mirror at run-time and perform integration tests. A similar approach can be done with unit tests.

Does this mean that performing tests while sending calls to the meteor server (ex: update my user’s role in a “Given I am an admin” step) is not supported ?

That’s supported and encourage :smile: You can combine DDP requests to server with browser calls as you wish.

What I was referring to was running Cucumber in a unit-test context

I am confused. Do the methods called from this.mirror.call need to be in fixtures or they can be regular Meteor.method functions from my actual server code ?

PS: Maybe this isn’t the best post to ask my questions in, if I need more help I’ll write up a new thread.

Sure thing, please do open another thread after this as it helps others find answers quicker in the future.

To answer your question, here are the different contexts you have to deal with:

The Cucumber Context - Here you have no direct access to any Meteor context. Cucumber runs in a separate node process. It gives you a DDP connection to the mirror

Mirror Server Context - You access this using this.server (or this.mirror) within the Cucumber step def context.

Mirror Client Context - You access this using this.client (or this.browser) within the Cucumber step def context. Webdriver allows you to execute code on the client using the methods this.client.execute and this.client.executeAsync

Main App Server / Client Context - You have no suported access to this context from within your tests or mirror code. This is intentional to keep complete separation between your app code and test code.

Fixtures - These are files that are copied from the /tests/cucumber/fixtures/ directory into the mirror only. This means you can add special endpoints that can fix data in your system. Putting a method in here that clears your database (like reset) will only be available in the mirror server context, and you can call it from the cucumber context using this.server or this.mirror.

Hopefully that clears it up!

It does. I still need practice but that’s the best explanation of the key concepts I could ask for. Thanks !

Hi all,

I went through a different posts in this forum, but, correct me if I am wrong, I couldn’t find the question I will post. I think it corresponds to this topic here.

I am new to javascript testing, and I have successfully added xolvio:cucumber to my project and did some features which run perfectly.

My question is regarding the setup of the type of reports I can get from Chimp, and how and where I can define this. Specifically, I would like to get the results from Xunit reporter, to have .xml files generated.

I tried in wdio.conf.js and wdio.cucumber.conf.js, but I didn’t manage to make it work.
Could you please help regarding this. Which files have to be edited in order to make this way of reporting work?

Hey

Cucumber does not have xunit reports out of the box, it produces json reports. You can translate these and I know there are a few initiatives to do that out there.

Also, we’ve just deprecated xolvio:cucumber in favor of chimp. You can see the migration guide here

Hello,

Tnx for the information. I did check a lot the Chimp page but didn’t notice the migration.

That’s because I added it after you posted :slight_smile:

Haha. Better late then never :wink:

OK so both this and velocity are now deprecated, in favor of chimp?

xolvio:cucumber is deprecated yes, and Chimp is the replacment for that package. Chimp is completely backwards compatible with the tests written in xolvio:cucumber, with the difference that Chimp does not have a dependency on Velocity.

Chimp is still the most comprehensive end-to-end testing solution for Meteor, and it now also works for non Meteor apps too.

We’re adding more and more capabilities to Chimp as an independent testing framework that has some extra support for Meteor.

Hopefully that clears it up :slight_smile:

2 Likes