Chimp Testing Documentation Logging and Predefined Steps

So far this looks like a winner. However, I’ve run into a couple of simple roadblocks. First: Is there a way to log to the terminal? Second, is there a set of predefined steps implemented (i.e., “when I fill in “this” with “that”” or "this I should see “this” within “that”? Finally, is there a list of ways to manipulate the preconditions. So for example,

Given I have the form fields "song name, artist"
And I fill in "song name" with "Bell Bottom Blues"
And I fill in "artist" with "Eric Clapton"
Then I should see "Bell Bottom Blues" within "td.song-name"
And I should see "Eric Clapton" within "td.artist"

Something along those lines.

Is there a way to log to the terminal?

Yes, with console.log(…) or anything else that outputs to stdout. If you use the Meteor package xolvio:cucumber, the log is in .meteor/local/log and you can watch it with tail -f .meteor/local/log/cucumber.log.

Regarding the other questions, you shouldn’t couple your specs with the UI. Sam has recently written about this in detail here: https://github.com/xolvio/automated-testing-best-practices/blob/master/content/TESTING-SUCCESS-FACTORS.md#the-key-lessons.

Oddly, I have a ./meteor/local but not a log directory. So I don’t know which bit-bucket my console.logs are going into. On the other subjects, testing is sort of a religious issue. I remember when Aslak was working on the Story Runner (precursor to Cucumber) and there was a lively discussion.

The whole idea of Cucumber was to automate running user scenarios. What the stakeholder expects to accomplish, the actions they see and the results they expect. The idea was (and to me, still is) that If I add a song using an in-page form, then the accumulated list of songs should contain that song. I get that changing the UI then breaks the build, but maybe it should.

But as I said, it’s an issue where nobody is right and nobody is wrong. It’s just important to do the tests the best way you know how. My question was to do with porting some Ruby-Cucumber knowledge over to Chimp.

Hey @sxross

First of all, it’s awesome that you’re writing tests :+1:

Any console.log that you do in your steps, they will be output to the console where the chimp results are output. If you scroll up a little, you should see them right next to the step status (red/green/yellow).

The rabbit hole is much deeper than a preference I’m afraid! Here is some material that I strongly recommend you read:

Tl;dr

  1. UI test scripts are technical debt, especially natural language ones
  2. The Gherkin parser cannot complete with software engineering
  3. The point of natural language parsing is domain discovery

I’ll be writing a section about how domain discovery is done soon.

If you insist on writing UI test scripts, I highly advise you to use Chimp + Mocha for those as you will have the breadth of software engineering practices to reuse code, which is far more powerful than the simplistic Gherkin parser.

Happy to help / debate :slight_smile:

You won’t get a debate from me. I’ve already read all three of your bullet points (great stuff, by the way) and will look at the example project on GitHub to see how you are accomplishing your goal. In my defense, the Gherkin steps I showed are not simply stated as shown; they are backed with Javascript (actually Coffeescript) code that does quite a bit more verification than a simple natural language expression that would be stakeholder-facing.

As you may see from Aslak’s post, his thinking has evolved significantly since he was involved in rSpec and even since he cooked up Cucumber (forgive the food pun). This is a fair amount to digest (again, forgive the food pun), as Cuke was always a great E2E solution as evidenced by screenshot verification and so on. I’ll read the material you’ve put out there and keep poking away at it.

In general though, I totally agree with your philosophy of getting the tests or “discoveries” in some verifiable state so you can tell when you break expectations. I’m trying not to reinvent the wheel. Again, referencing Aslak, when he initially released Cuke, he provided some steps that worked with a few Web drivers. He has since removed these from the distribution because he considers them an anti-pattern. As you may infer, I disagree to some extent, as I prefer to write less code rather than more and simple exceptions like “I should see” don’t seem like stuff I should need to code (although I have no problem doing so).

Thanks to both you and @sanjo for the detailed replies and please know that they are highly useful and appreciated.

1 Like

Glad you find this useful, and I find your feedback useful too because every thread like this one, helps me build up content for the automated testing best practices repository we’re building.

What I really want is that developers to truly know what they’re getting in to through a proper understanding of the problem, rather than someone just preaching to them what should and shouldn’t be done. So I hope the feedback below is good for you personally, and the next person that reads this thread.

To your point about:

I couldn’t agree more with good reuse, and here’s how I would do it with Gherkin. Consider this feature:

#Feature: Account holder same-bank money transfers

#  As an Account Holder
#  I want to transfer money from my bank account to other account holders in the same bank
#  So that I can pay my dues

#  @critical
#  Scenario: Account has sufficient funds
#    Given my bank account balance is $100
#    And "James"'s bank account balance is $0
 #   When I transfer $20 to "James"'s bank account
    Then my account balance should be $80
#    And "James"'s account balance should be $20

Notice how “Then my account balance should be $80” contains a reference to the account domain model that contains the property balance with the value $80, where as a reusable “Then I should see $80” does not contains any domain context.

Here’s how I would do code-reuse. The step definition for “Then my account balance should be $80” looks like this:

this.Then(/^my account balance should be \$(\d+)$/, function (balance) {
  var actualBalance = server.execute(function () {
    return AccountHolders.findOne(Meteor.userId()).account.balance;
  });
  expect(actualBalance).toEqual(parseFloat(balance));
});

That’s purely testing the domain portion. We can also define a step for the UI testing portion:

this.Then(/^my account balance should be \$(\d+)$/, function (balance) {
  var actualBalance = widgets.accountSummary.getBalance();
  expect(actualBalance).toEqual(balance);
});

And now we have as much reuse as we like. Any time we need to test the account balance through the UI, we can use the accountSummary widget, and any time we want to test the account balance through the domain, we use the domain directly.

For completeness, here’s the accountSummary widget code:

function AccountSummaryWidget() {
  var selectors = {
    balance: '.account-summary__balance'
  };

  this.getBalance = function () {
    return browser.getText(selectors.balance);
  }
}
widgets.accountSummary = new AccountSummaryWidget();

As a side note, it’s really interesting how well the BEM syntax plays TDD’ing UI components. I was able to build out a React components from this test.

There is a real pearl of wisdom in here. It’s excellent at a conceptual level to talk about testing domain, integration, e2e, whatever you like, but here is a concrete example:

this.Then(/^my account balance should be \$(\d+)$/, function (balance) {
  var actualBalance = server.execute(function () {
    return AccountHolders.findOne(Meteor.userId()).account.balance;
  });
  expect(actualBalance).toEqual(parseFloat(balance));
});

There is syntactic sugar associated with assigning the result to a variable (actualBalance), but the key to me – the aha moment, if you will, is the idea of using server.execute in the definition of a cuke step.

So, thanks for that. This is a departure from how most Rubyists use Cucumber AFAICT. But it makes sense both in terms of making tests faster and in terms of beefing them up against UI changes.

I’m still trying to get a test harness that’s easy to start. I can start Chimp manually and that works fine, but you’ve got that cool script and I’m still puzzling out how that works and how to adapt it to my project.

Fantastic to hear. Not to forget that Chimp also allows you to do the same thing using Mocha:

describe('Bank Service', function() {
  describe('Same bank transfers', function() {
    it('should adjust the balance of the account holder and the recipient', function() {
      server.execute(function() {
        // SETUP
        var accountHolder1 = new AccountHolders({'account.balance': 50 });
        var accountHolder2 = new AccountHolders({'account.balance': 0 });
        accountHolder1.save();
        accountHolder2.save();
        // EXECUTE
        BankService.transfer(accountHolder1.get('_id'), accountHolder2.get('_id'), 20);
        // VERIFY
        accountHolder1.reload();
        accountHolder2.reload();
        expect(accountHolder1.get('account.balance')).to.be(30);
        expect(accountHolder2.get('account.balance')).to.be(20);
      });
    });
  });
});

Gotta love Astronomy!

I actually prefer Cucumber for domain discovery as it’s more reusable. If you think about it, separating out the Given (Setup), When (Execute) and Then (Verify) allows you to test the domain from different directions.

Thanks for letting me know that, I’ll add a lifecycle flowchart to explain how it works.

you used same gherkin string for UI and non-UI
what is the best convention you use to differentiate between these 2 cases which sounds good and enable you to use these naming conventions consistently in whole app ?

And do you recommend using server side kinda function/method calls or emiting events in simulations ?
The event emitter events sounds quite good to me as a way to also define kinda API for communication between packages. Or good way how to hook optional packages to business logic. In case we will be breaking app to microservices in future.

Great question!

The convention is directory based. If you look in the directory structure in the automated testing best practices repo, you’ll see there is a complete set of step definitions under the domain directory, and there are a few step definitions in the critical directory.

We recently added a feature to Chimp that tells it to run twice. Once for the default untagged scenarios (the domain) and another run for the @critical tagged scenarios (the UI). To enable this mode, you pass Chimp the --criticalSteps option.

The --criticalSteps option is a path that tells Chimp where to find the steps to prioritize when running @critical scenarios. If there are overriding steps, they will be used, if there aren’t overriding steps, then the default steps will be used.

The Workflow
We always start with the domain, and once we have a clear understanding of the domain through genuine coding iterations, then we choose a single critical scenario per feature to automate UI tests for.

TDD is used throughout. First we drive through the domain, and every time we write any unit of code, we drive with a unit test.

After the single critical scenario works, we expand the UI component functionality to cover the non-critical steps with unit tests only.

This is how you get the perfect balance of UI, Service and Unit tests.

Let me know if that explanation made sense!

I think it depends on what your app architecture choices are. I personally like the dependency injection approach, where you ask a container for services and call them directly. In our best practices repo, the services are currently global. This is also a (semi) practical option since services are singletons. You may want to use a service discovery pattern of sorts of you are using microservices.

Consistency is key here. Whatever service access mechanism you use in your production code, is the same mechanism you should use in your simulations.

What’s interesting here is that the service access mechanism is not central to the tests and becomes much easier to refactor, so you don’t have to get it 100% right to start. As long as you focus on the domain and units, then these sorts of refactors can happen with the full confidence that the entire codebase will be safe from regression bugs!

Some kind of chart and overview of the necessary files and configurations would indeed be great. Having looked through the repository I’m still having a hard time figuring out which parts are relevant or necessary for the setup to work, and which things to change, since a lot of it is integrated with the app.

Here’s what happens when you run npm start

I’ve also described what happens when you run npm test too, so be sure to check it all out on the the updated README for the scripts.

Please let me know what’s missing so I can improve it.

I have one question regarding the hooks. I am successfully using the After hook, but I need to perform some actions after all scenarios finish executing.
Is there some way to accomplish this in an easy manner?

Yep, check these events out:

https://github.com/cucumber/cucumber-js/blob/master/lib/cucumber/listener/events.js#L8-L10

I missed it. Great, tnx very much for the fast response!