Testing Meteor apps in 2018
After almost 2 years with Meteor, there is one thing that just keeps staying hard, unreliable, and really obscure: testing Meteor apps.
Disclaimer: Probably almost all of this and my issues is because I’m still a young engineer, but I feel it still doesn’t justify the level of absolute hair-pulling/ripping experience I’ve experienced so far. I apologize in advance for hurt feelings, I’m a bit emotional right now as well.
There is large amount of test driver packages:
chimp, tried it once, was excessively messy, then quit
Each of these have subtle, unclear differences (I use 3 of these 8 for various terrible reasons, some of which I can’t even remember).
- Some tests pass with one driver, others don’t
- Some tests fail with one driver, others pass
- Sometimes they support all mocha features, sometimes they don’t (like
- Sometimes they run in the browser, sometimes they only run in the console
- Sometimes they watch tests properly, sometimes they don’t
- Sometimes they crash with one headless browser, sometimes they don’t (I’ve tried them all, chrome, nightmare, spacejam, electron, and others I can’t remember)
- Sometimes they run client and server tests, sometimes they only run server tests
Which one is recommended today?
meteortesting:mocha. Which one will be recommended tomorrow? Nobody knows. Right now
meteortesting:mocha doesn’t seem to even be very actively maintained.
There are a bunch of packages that are supposed to help with writing tests:
dburles:factory: cool but not always reliable. Lacks powerful relational collection tools
xolvio:cleaner: Does its simple job (or at least I think)
fakerJS: Cool stuff
hwillson:stub-collections: Avoid if possible (more below)
And of course the usual, that seem to just work:
I also want to mention that I tried to use
jest at some point (by following this tutorial) to get access to snapshot tests on the client, but Meteor really didn’t like jest (can’t remember the specifics), so no snapshot tests for me at the moment (which makes me sad). Did someone figure out how to do this ?
Mocking Meteor packages
This has given me nightmares as well, but I finally have a somewhat working solution to mock my meteor packages so that I can use normal (fast) test runners. (tip below)
I currently have 2 different types of unit tests:
spec tests have 0 dependencies to meteor, so that I can use WallabyJS (really awesome). And the
test tests have all the messy meteor tests that I can only run with
meteor test and which take 5-10 seconds to reload (which is an unacceptable amount of time to me in the era of near-instant webpack reloads). I try to avoid writing meteor tests as much as I can, because it just makes your time melt like hot butter, I hate doing it, which in itself is really really bad because it reduces the quality of my work.
It’s an immense pain that meteor packages are hidden away by Meteor and can’t simply be used by normal test runners instantly, I wish all packages at least were consistently accessible in
.meteor/local/packages/ for example. So at least I can point to them with babel plugins etc.
build folder with the relevant packages doesn’t seem to always appear properly, I don’t know how to debug my tests when they fail because of a package either. (I could write a whole other post on meteor debugging, another bad developer experience with VSCode for example. The fact that
meteor debug only has 20 lines of documentation is pretty sad).
Meteor packages and mocking them feels like a gigantic hack to me that does not add confidence to the development experience.
Other Meteor issues
- Setting up tests in one file that runs a bunch of preparatory code is not trivial, things get imported in unclear ways, sometimes my tests ran before other code was executed. Mind-bending tutorial here (does it even apply to
meteor test? maybe, but you have to mix that with this documentation, now… what loads when?)
- Sometimes Meteor collections just do not work issue here and another here
- I’ve recently started experiencing this again:
MyCollection.findOne()just returns undefined completely randomly, which was so infuriating that I decided to write this post.
- I’ve recently started experiencing this again:
- Some issues may arise because the testing packages are not ordered properly in
packages, this has sometimes solved some extremely dark super-hard-to-debug things. To me it’s still pure magic, and really scary. I don’t understand why I have to deal with that in Meteor and not in any other framework.
- I still can’t seem to log in on the server to test things properly (issue here). Now a Meteor employee recommends I start writing hacky functions with the
DDPpackage just to write a single test? Why is his recommendation not documented anywhere?
jsdomin a meteor app to run client tests on the server (as you should for a SSR app for example) crashes production deployments issue here
We use circleCI to run our tests, and the experience there has been flaky at best as well. Random meteor timeouts, difficulty managing different meteor versions and globally installed packages. Huge initial build times when the cache is cleared (>10-15 mins just to print
App running at: localhost:3000), which then drops to just 2 minutes in a second run (insane difference…?) with caching.
If I want to run tests on my app I have to wait these 2 minutes for meteor to start up, and then my tests run in less than 10 seconds. My entire build (3 apps, end-to-end and unit each) currently takes roughly 25 minutes, but could easily take less than 10 if I wasn’t always waiting on Meteor.
I know you can’t have it all, and Meteor certainly sped up some things, but from what I hear about my peers who didn’t make the (sometimes I really feel is a mistake) choice to go with Meteor, I feel like something has got to improve.
Meteor testing manual
Honorable mention to this project by the great Sam Hatoum, which made me really excited 2 years ago, sadly it appears to have been difficult for him to get it done: https://www.qualityfaster.com/landing/meteor-testing.html
His last update on his website is promising though and I really hope he can make it through!
As a result I write as little meteor tests as I can because the developer experience is so bad. I hope this is only due to my inexperience and lack of understanding of Meteor testing, but I’m surprised how little I see on here about writing better tests. As if most people either: 1) Don’t write tests (understandable with Meteor being mostly used for prototyping), 2) Figured everything out but don’t have the time to share (I totally understand that), or 3) Are as lost as I am and live with it but power through with terrible tests 4) Are as lost as I am and don’t write many tests
I do have a few things to share that I learned (I wish there was more):
- Don’t use
hwillson:stub-collections. I was all in at first, because it’s recommended everywhere and I was naive. It speeds up your tests, but my 500 tests run in 7 seconds, so I really don’t need it. And most importantly it mocks your collections, all the packages that modify them (like
collection2, open issue here), which makes you write tests that don’t actually test your real world app. I think this is a very bad pattern that you should avoid as much as possible, except if you really know what you’re doing. (hint: I don’t)
- Put all your meteor methods code in a separate
classwhich is itself called in meteor methods. This was recommended by @diaconutheodor in his great
meteor-tuts.com. Meteor methods should only make a security check and then call another class/Service.
- Mocking meteor packages: I use babels
module-resolverplugin, and then assign
meteorto a folder I called
meteorStubs, where there is a file for each package:
- One issue I’ve encountered, is that Windows based developers can’t use this because “:” is disallowed in their file system. Luckily we’re all on Mac OS
- We’ve successfully started using Cypress.io for our end to end tests. It wasn’t completely easy, but we now have a pretty nice setup. An example of which you can see here
- Talking about
.meteor/packagesorder, make sure to place
dburles:factory. Or else be prepared to have violent sweaty nightmares about weirdly failing tests
- Use WallabyJS if you can afford it, and use my config as a starting point if it helps. Find it here. This config was actually improved by Artem Govorov from the Wallaby team. He had to dig inside the meteor internals to figure out which babel plugins meteor was secretly using.
I really do not feel confident in my meteor testing skills to push some of this forwards. I wish more was shared and written about really solid testing practices with meteor by senior devs/engineers, that have a good developer experience and can push more people to write more tests.
I’m pretty sure that all the issues I have are hidden somewhere in my code, which is a result of jumping from multiple driver packages, seeing best practices change, and meteor evolving and improving, etc.
I failed to help the community as well by not digging deeper and trying to create reproductions of my issues, but after spending days fighting with an issue and putting my own business in trouble repeatedly, I really can’t justify spending another several hours to extract the weird issues I’m encountering. I just don’t have the ressources. I really wish I did and I could help more. For this I’m sorry, as that makes this post a bit weaker.
These issues really make me regret going with Meteor every now and then, because I’m now very confident that the initial speed boost I got when starting has been easily cancelled out by these endless days figuring out why the meteor magic does what it does (or doesn’t). While the rest of the developer community just moves forwards with widely supported, reliable, well-documented tools, better developer experiences and overall just getting shit done. Spending days figuring out how to write a few meteor tests is preventing me from shipping things.
What I secretly wish for
Here’s a couple things that would make me sleep better:
- Near instant test reloads (and regular app reloads as well for that matter)
- Proper production-ready testing documentation with all the tricks and tips to write great, scaleable tests for all sorts of use-cases. Given the size of the Testing section (which I will say has improved), testing just really feels like a total afterthought with Meteor.
- Snapshot tests ?
- Visual regression tests ?
- A bigger budget at Qualia to write more awesome advanced meteor tutorials @veered
Please let me know if any of you is also experiencing this frustration, or if I’m alone with this