If I’m understanding things correctly is there still a bit of an issue here, not with the solutions above but in general: It seems like we’re saying it’s fine for the unit tests or the code being tested to import npm modules and use their functionality in the system under test, but you can’t do the same with the meteor modules, instead you have to stub them out?
That makes good sense where those Meteor modules are for things that shouldn’t be tested as part of the code you are unit testing (routing, sending and receiving data from the server etc); it makes complete sense to stub those modules out.
However, there are some Meteor modules that I do want to be tested as part of my unit tests, I’m happy to accept them as black boxes and I want my unit tests to fail if those modules change in some way after I update them.
One example of that is underscore; I don’t want to have to stub out underscore, and even if I did the code I was testing would then stop working unless I faked underscore’s behaviour in some way. I could get round this by using underscore from npm, but then I would be sending two versions of underscore to the browser (I think, because Meteor uses underscore too?). And really, that’s a bit of a hack as effectively now I’m saying saying “it’s OK to use this if it comes from npm, but not from Meteor” which is then a decision based on implementation rather than the requirements for the test.
Essentially, I guess there are Meteor modules that I want to treat like JavaScript’s built-in functions (I wouldn’t stub out toExponential say).
I think your reasoning is fine and there are two solutions available to your problem today:
Use the Meteor 1.3 test runner (meteor test)
Use the Node.js code from the Meteor boot.js file to load the Meteor packages into your Node.js testing process. I use this technique and have roughly outlined it in my previous post.
This is one of the symptoms of the separate Meteor package system. It is hard to use Meteor packages in a Node.js context. But Sashko and Zoltan have indicated that they are aware of this issue and Meteor might use NPM in the future.
@Sanjo I’ve been playing with your wallaby_server.js file, and after some struggles I got it to work.
One question though, are you aware that you must use Fibers within your tests? And do you know a way around this?
For example when I try to reset the database in a beforeAll hook, I get the most famous:
“before all” hook: Meteor code must always run within a Fiber. Try wrapping callbacks that you pass to non-Meteor libraries with Meteor.bindEnvironment.
Running the code all in a fiber solves the issue, but I don’t like stuffing up my tests with fibers. Do you know of a way so Mocha does this for you?
Thanks @Sanjo, JS testing is new to me (.net background) so I’m still getting up to speed on all of this.
Just to double check, do you mean use the 1.3 test runner separately without Wallaby, or is it potentially possible to set Wallaby up to use meteor’s test runner?
Also, would that approach then mean waiting for a Meteor build to see the results of the code changes? That’s what I’m keen to avoid, I’m trying to get to the super fast red-green feedback that I have in nCrunch or when using wallaby on pure JS classes outside Meteor.
If you want to use Wallaby, the Meteor test runner (meteor test) is not an option.
We have what you want working, but it’s not super beginner friendly yet. It lacks some detailed documentation and abstractions to make the code easier to use. Check out what I wrote recently in https://github.com/xolvio/automated-testing-best-practices/issues/27 and also just clone and try the repo. Then try to copy the necessary files and adjust the Wallaby configs to your needs.
If you are willing to pay money, we can also help you with our paid Xolv.io support. I will make the code more easier to use eventually, but currently I have a high priority on a new product, that we develop
Thank you all for your work on this. Have any of you had any luck using testdouble with Wallaby? I tried @sanjo’s wallaby_server, took a bit of reconfig for Mantra, but I never got it to successfully work loading meteor packages.
Yes, wallaby is the reason I started to look into this topic. If you follow the steps in my post above, or the blog post of @pcorey, wallaby will work just fine.
Read this post carefully, note the reset and the require part.
About the wallaby_server you’re referring to. It can be done. It needs some reconfiguration, but then you’re able to run Meteor inside your IDE with full wallaby support. It is however way slower than stubbing some methods out like described above. So I wouldn’t recommend this.
The idea of wallaby is instant feedback. Not to wait 1 to 3 seconds for something to happen.
I’ve finally taken the time to get external Meteor unit testing and wallaby (via Atom) all wired up. I’ve tried using both approaches discussed in this thread - 1) mocking all Meteor dependencies with something like testdouble, and 2) loading the Meteor context to leverage Meteor dependencies (and Meteor packages) directly. Here’s a quick breakdown of my findings.
1) Mocking Meteor Dependencies
Going with the mocking all Meteor dependencies approach definitely returns testing results faster. That being said I’ve found that my unit tests quickly become a big ball of dependency mocking and injecting, making it difficult to track down the true nature of certain tests. I’ve gone back over some complex tests after doing this and had to take a few minutes to figure out what I was doing. Yes I always try to refactor my code to keep dependencies to a minimum, but life is a balancing act between time & money for that time :-).
Mocking all external dependencies can be a real pain when dealing with certain packages. Take mdg:validated-method for example - when creating a new ValidatedMethod object, the core of the functionality for that object is passed in as a constructor param. This causes grief when it comes to mocking with testdouble. To work around this I had to create a ValidatedMethod wrapper class that exposes ValidatedMethod internals, to make testing easier. Another example - working with aldeed:simple-schema; I was testing some code that makes extensive use of Simple Schema’s validation API, which means I had to mock several aspects of Simple Schema out. This added a lot of text complexity and extra overhead, which really took time away from putting the focus on what I was supposed to be testing.
With testdouble specifically, I ran into issues where mocked dependency changes weren’t being picked up between tests. I tracked this down to my use of require and Node’s require cache. I had to add something like the following to my test tear downs to get back to back tests to pass in certain cases:
2) Loading Meteor Dependencies (Core and Packages)
To get this running, I followed / modified the approach @sanjo/@sam put together in the awesome automated-testing-best-practices repo. I now have full client/server unit tests running with all Meteor dependencies in place.
With this approach I can choose to mock out meteor/* imports if I want to, or leave them in place if I just want the default code to run - super nice!
I’m using Wallaby with Atom, which unfortunately means (for now) that I’m limited to running all client tests in one Atom window, and all server tests in another. I thought this would be a bigger deal than it really is though - I actually don’t mind keeping 2 separate editor windows open running different sets of tests.
I really like using testdouble but had a hard time finding an approach that let me properly leverage testdouble’s td.replace syntax when importing meteor/* based packages, on both the client and server. I chose to work around this by changing my meteor/* based imports to look like:
import { Meteor, ValidatedMethod } from '../../utility/meteor/packages';
With this in place I can properly mock all import meter/* package calls via testdouble (on both the client/sever), when I want to.
Summary
It took a bit of sweat, but I’m really glad I invested the time to get this all working. I’m getting MUCH faster unit test feedback now, which is awesome! I’m currently leveraging both approaches on different projects, but for the larger full web-apps I’m working on (not packages), I’m using the second load the meteor context approach. Sure my feedback times are a bit slower with Wallaby, but we’re talking an average of 1-2 seconds for the test/code feedback loop to complete, versus the 7-9 seconds I was seeing with meteor test. I’m also running this on a 4 year old macbook air, so your mileage will likely be much better!
A big, BIG thanks to everyone that has been looking into this and sharing their findings!
That’s really, really helpful @hwillson, many thanks for writing it up. I’ve been working on this again this week too so this is really well timed for me.
It’s reassuring to see someone more experienced also run up against the issues I’d been having with approach 1).
I’m interested in what you needed to change in approach 2 and also did you get chimp running too?
I’d really like to see the Guide cover approach 2 and for that to become an officially supported solution. I’m nervous about going down that path alone as I know I don’t have the knowledge yet to maintain those files if things break (which seems likely as I’m developing on Windows and so things like simlinks aren’t available). I’m beginning to think the best approach may simply to be to switch to a Mac for Meteor dev (although that will be a nuisance for my .net work potentially).
Adjusted the babel settings a bit (removed stage-0 and transform-decorators-legacy since I don’t need them)
Removed the config items for handling package stubs (since I’m handling this differently)
Adjusted the files and tests sections to match my project’s directory structure
Added instrument: false to all loaded Meteor files to avoid code coverage instrumentation on those files
Switched testFramework to mocha
Disabled debug
In the __meteor_runtime_config__.js file I added to my projects, I changed the NODE_ENV setting to just development (to work around an error I was getting)
Hard coded my home directory at the top of the file to get around HOME: unbound variable errors I was getting (I was thinking of submitting a PR that uses os-homedir for this instead but haven’t had a chance yet)
Adjusted the files and tests sections to match my project’s directory structure
Switched testFramework to mocha
Disabled debug
I haven’t tried yet - it’s on my list though.
Looking over the number of Windows related issues that have been coming up on the forums over the past while, this (or Linux) might be the best choice unfortunately …
All of that is really useful, many thanks. I’d made several of those changes too (under guidance from @sam). I’m going to try and look at approach 2 again over the weekend and see if I can get comfortable with it as it I think it really is the preferable approach as you can still mock things out anyway.
I really wish I could get this working, but I’m unable. My problem is that there’s a chain of imports going on that eventually imports meteor/mongo, and proxyquire is… not doing its job? Example:
tests/ChipInput-test.js
import React from 'react';
import { expect } from 'chai';
import { shallow } from 'enzyme';
import proxyquire from 'proxyquire';
import sinon from 'sinon';
const Mongo = {
Collection: sinon.spy(),
};
proxyquire('../lib/collections/tags.js', {
'meteor/mongo': { Mongo, '@noCallThru': true },
});
// This file imports a file which imports another file which imports '/lib/collections/tags'
import { ChipInput } from '../client/components/Chips/ChipInput';
describe('<ChipInput />', _ => {
xit('needs tests', done => {
done();
});
});
And sadly, when I run npm test, I still see this:
Error: Cannot find module 'meteor/mongo'
at Function.Module._resolveFilename (module.js:438:15)
at Function.Module._load (module.js:386:25)
at Module.require (module.js:466:17)
at require (internal/module.js:20:19)
at Object.<anonymous> (tags.js:1:1)
[...]
What file is depending on meteor/mongo here? Because you’re importing tags.js, but you don’t seem to use this one in your tests? Currently you rewire tags.js but not the ChipInput component.
I’m not sure, but perhaps is this more like what you need?
The file depending on meteor/mongo is /lib/collection/tags.js. I tried your code above, and unfortunately it didn’t work, same error as before. ChipInput imports ./Chip which imports chipTypes from ./index which then does:
import { Tag } from '/lib/collections/tags';
and tags.js starts out with:
import { Mongo } from 'meteor/mongo';
import { Class } from 'meteor/jagi:astronomy';
[...]
So I have no idea how to use proxyquire to follow this chain of imports and somehow stub meteor/mongo and any other Meteor packages it comes across.
Cross-posting from the thread Unit Testing with React where I used @smeijer’s testdouble/require know-how to get a very minimal POC for enzyme testing the todos app:
It’s really hideous right now, but if anybody has suggestions on how to clean it up, please comment in the PR and I’m happy to bang on it some more.
