CucumberJS and PioneerJS

testing
1

This topic is a continuation of another discussion here.

@codingadventure
I’m interested in how running Cucumber/Pioneer in the Meteor context has worked for you in space:pioneer

I first had xolvio:cucumber this way. I had a mechanism that monkey patched the callbacks in the steps so they were wrapped with Meteor.bindEnvironment. this generally worked but some steps were a little unruely, like ‘background’ and ‘before’ were inconsistent in their parameters as new cucumber releases came out. It was all manageable, but the real gotcha was the module caching. As much as I cleaned the module cache, I still found a few lingerers!

The solution I settled on was to move Cucumber to run entirely outside the app, but autowired a DDP connection that automatically connects from the world to the app/mirror. This, plus the ability to have test-only files (fixtures) in the mirror has ended up much cleaner without having to monkey patch Cucumber.

Are you doing the same / similar thing with Pioneer?

1 Like
2

Hey Sam,
thanks for moving this discussion over here :wink:

I had no issues with module caching etc. – the only thing that was important for pioneer.js is that the @meteor method returns a promise which eventually resolves or rejects. Other than that, I can work in their as normal with Meteor.

The package space:pioneer uses pioneer.js / cucumber-js under the hood and simply patches the cucumber world, so that you have a method on the context that you can call to work in the Meteor environment. This makes it possible to deeply integrate with your app, which is invaluable to speedy acceptance tests where you don’t have to click and point through each and every step.

Here is a good example of how that’s useful:

@Given /^I open the project "([^"]*)"$/, (projectTitle) ->

  @meteor (resolve, reject) ->
    project = Prisma.Projects.findOne title: projectTitle
    if project? then resolve(project) else reject new Error 'No project found'
  )
  .then((project) => @driver.visit "#{@rootUrl}/projects/#{project._id}")
  .then => new @Widgets.SwatchList().shouldBePresent()

This step definition allows you to open a project (within my app) just by providing the title, which is very convenient when writing the feature files because I don’t have to throw ids around.

So for me it would not be sufficient to only connect to my Meteor app via DDP. This would make everything more complex and I would have to maintain an extra (public!) API to do all the little things necessary during testing.

There is a lot more than I showed in the example above. Here is a snippet from my cucumber configuration, that ensures that emails are never really sent during testing and collected for further inspection when I add the email tag to any scenario:

@Before '@email', -> @meteor (resolve) =>
 @EMAILS = prisma.injector.get 'Email'
 emailSendBackup = @EMAILS.send
 @EMAILS.sentEmails = []
 @EMAILS.send = (options) ->
   console.log "-> sending email <#{options.subject}> …"
   @sentEmails.push options
 resolve()

@After '@email', -> @meteor (resolve) ->
  emailService = prisma.injector.get 'Email'
  emailService.send = emailSendBackup
  delete emailService.sentEmails
  resolve()

and later in a specific scenario:

@Then /^I should receive a welcome email$/, ->
  @meteor (resolve, reject) => Meteor.defer =>
    email = @EMAILS.sentEmails[1]
    if email? and email.subject is 'Welcome to Prisma!'
      fulfill()
    else
      reject 'Did not receive welcome email'

How would you do that with the mirror api + DDP connection without introducing a public api to stub your emails service?

3

I had meteor-cucumber previously be setup in a similar way where it ran inside the app, but I later moved it to be outside. From a philosophical point of view, I like to keep galvanic isolation between test-code and the system under test (SUT). Mainly because the tests should act like the consumer acts on the SUT. But I understand that everyone has their preference of course!

I really like the Widgets in Pioneer and Sanjo and I are talking about borrowing them :slight_smile: We already do far too much stuff with starting phantom/selenium/saucelabs/appium to move away from our cuke-monkey library, so we’ll see if there’s a good integration point. Another option I was considering lately is to allow the user to specify the cucumber runner, so they can use cucumberjs or pioneerjs and still get the benefits of cuke-monkey.

I totally agree that you shouldn’t have to click on each and every step and you need to quickly get setup to be able to test the thing you’re interested in.

There are two good patterns we use for keeping test code private. The first is using fixture files. These files are loaded into the mirror only by Velocity from the /tests/FRAMEWORK/anything-fixture.(js|coffee). This pattern is the quick and easy option, but there’s a far more powerful pattern which is to use debugOnly packages. These will not get bundled by Meteor which means they stay out of production, but still have full access to your app. See here.

By using these fixture packages, the same email mocking code you have above can be used by your Jasmine tests as well as your cucumber test, which is a genuine use-case for DRYing up your test code.

To answer your question, here’s how I would do what you’ve done with a complete galvanic isolation:

The package xolvio:inbox-stub is a debugOnly package that overrides the Email.send like you have (but only in the mirror) and provides DDP endpoints for control. See the code here.

So after you add the package, you can now in your hooks say something like:
disclaimer: I’m only using coffee to be nice so don’t judge me, I never use coffeescript!

@Before '@email', ->
  @mirror.call 'resetInbox'

and in your scenarios:

@When /^I enter my email and click subscribe$/, -> 
   @browser.type('#email', 'hello@there.com').click('#subscribe')

@Then /^I should receive a welcome email$/, ->
  @mirror.call('getEmailsFromInboxStub').should.eventually.contain 'Welcome to Prisma!'

Isn’t that clean? :wink:

With meteor-cucumber we actually a promisified both cucumber and the node DDP client, and baked in chai-as-promised, this allows us to do cool stuff like the syntax above without callbacks. Just returning the mirror.call is enough.

Of course you can also chain the mirror.call with another method that uses the returned objects directly so that you don’t have to throw ids around.

The combination of mirror DDP + fixture packages pattern + promises everywhere = a great mix for clean reusable test code.

I’m super happy that you want to integrate with Velocity, I’ll contact you privately and we can have a hangout to discuss.

4

Hey @sam!
Wow, that looks awesome, maybe its time for me to switch back :wink:

But like you said, I really love the pioneer.js widget concept – although I am not totally happy with some implementation details and think we could do a much better job. So maybe its better to continue that path, improving your package, instead of re-writing all that stuff for space:pioneer!

I think i might just start writing all new acceptance tests with meteor-cucumber to test-drive your suggestions and get a feeling for the package. In parallel I can help you implement the MVP for pioneerjs-like widgets so that I can port my other tests over to our implementation.

1 Like
5

That would be amazing! The community would really benefit from our combined efforts. Let me know about your experience with swiching to meteor-cucumber

2 Likes