Testing Meteor apps in 2018, a rant

florianbienefelt · July 20, 2018, 10:57am

/rant on

Testing Meteor apps in 2018

After almost 2 years with Meteor, there is one thing that just keeps staying hard, unreliable, and really obscure: testing Meteor apps.

Disclaimer: Probably almost all of this and my issues is because I’m still a young engineer, but I feel it still doesn’t justify the level of absolute hair-pulling/ripping experience I’ve experienced so far. I apologize in advance for hurt feelings, I’m a bit emotional right now as well.

Package overload

There is large amount of test driver packages:

dispatch:mocha
cultofcoders:mocha
practicalmeteor:mocha
meteortesting:mocha
tmeasday:acceptance-test-driver
mike:mocha deprecated
https://github.com/arunoda/laika deprecated
chimp, tried it once, was excessively messy, then quit

Each of these have subtle, unclear differences (I use 3 of these 8 for various terrible reasons, some of which I can’t even remember).

Some tests pass with one driver, others don’t
Some tests fail with one driver, others pass
Sometimes they support all mocha features, sometimes they don’t (like .only)
Sometimes they run in the browser, sometimes they only run in the console
Sometimes they watch tests properly, sometimes they don’t
Sometimes they crash with one headless browser, sometimes they don’t (I’ve tried them all, chrome, nightmare, spacejam, electron, and others I can’t remember)
Sometimes they run client and server tests, sometimes they only run server tests

Which one is recommended today? meteortesting:mocha. Which one will be recommended tomorrow? Nobody knows. Right now meteortesting:mocha doesn’t seem to even be very actively maintained.

There are a bunch of packages that are supposed to help with writing tests:

dburles:factory: cool but not always reliable. Lacks powerful relational collection tools
xolvio:cleaner: Does its simple job (or at least I think)
fakerJS: Cool stuff
hwillson:stub-collections: Avoid if possible (more below)

And of course the usual, that seem to just work:

chai
mocha

I also want to mention that I tried to use jest at some point (by following this tutorial) to get access to snapshot tests on the client, but Meteor really didn’t like jest (can’t remember the specifics), so no snapshot tests for me at the moment (which makes me sad). Did someone figure out how to do this ?

Mocking Meteor packages

This has given me nightmares as well, but I finally have a somewhat working solution to mock my meteor packages so that I can use normal (fast) test runners. (tip below)

I currently have 2 different types of unit tests: *.spec.js and *.test.js. The spec tests have 0 dependencies to meteor, so that I can use WallabyJS (really awesome). And the test tests have all the messy meteor tests that I can only run with meteor test and which take 5-10 seconds to reload (which is an unacceptable amount of time to me in the era of near-instant webpack reloads). I try to avoid writing meteor tests as much as I can, because it just makes your time melt like hot butter, I hate doing it, which in itself is really really bad because it reduces the quality of my work.

It’s an immense pain that meteor packages are hidden away by Meteor and can’t simply be used by normal test runners instantly, I wish all packages at least were consistently accessible in .meteor/local/packages/ for example. So at least I can point to them with babel plugins etc.

The build folder with the relevant packages doesn’t seem to always appear properly, I don’t know how to debug my tests when they fail because of a package either. (I could write a whole other post on meteor debugging, another bad developer experience with VSCode for example. The fact that meteor debug only has 20 lines of documentation is pretty sad).

Meteor packages and mocking them feels like a gigantic hack to me that does not add confidence to the development experience.

Other Meteor issues

Setting up tests in one file that runs a bunch of preparatory code is not trivial, things get imported in unclear ways, sometimes my tests ran before other code was executed. Mind-bending tutorial here (does it even apply to meteor test? maybe, but you have to mix that with this documentation, now… what loads when?)
Sometimes Meteor collections just do not work issue here and another here
- I’ve recently started experiencing this again: MyCollection.findOne() just returns undefined completely randomly, which was so infuriating that I decided to write this post.
Some issues may arise because the testing packages are not ordered properly in packages, this has sometimes solved some extremely dark super-hard-to-debug things. To me it’s still pure magic, and really scary. I don’t understand why I have to deal with that in Meteor and not in any other framework.
I still can’t seem to log in on the server to test things properly (issue here). Now a Meteor employee recommends I start writing hacky functions with the DDP package just to write a single test? Why is his recommendation not documented anywhere?
Installing jsdom in a meteor app to run client tests on the server (as you should for a SSR app for example) crashes production deployments issue here

CI

We use circleCI to run our tests, and the experience there has been flaky at best as well. Random meteor timeouts, difficulty managing different meteor versions and globally installed packages. Huge initial build times when the cache is cleared (>10-15 mins just to print App running at: localhost:3000), which then drops to just 2 minutes in a second run (insane difference…?) with caching.

If I want to run tests on my app I have to wait these 2 minutes for meteor to start up, and then my tests run in less than 10 seconds. My entire build (3 apps, end-to-end and unit each) currently takes roughly 25 minutes, but could easily take less than 10 if I wasn’t always waiting on Meteor.

I know you can’t have it all, and Meteor certainly sped up some things, but from what I hear about my peers who didn’t make the (sometimes I really feel is a mistake) choice to go with Meteor, I feel like something has got to improve.

Meteor testing manual

Honorable mention to this project by the great Sam Hatoum, which made me really excited 2 years ago, sadly it appears to have been difficult for him to get it done: https://www.qualityfaster.com/landing/meteor-testing.html

His last update on his website is promising though and I really hope he can make it through!

Current situation

As a result I write as little meteor tests as I can because the developer experience is so bad. I hope this is only due to my inexperience and lack of understanding of Meteor testing, but I’m surprised how little I see on here about writing better tests. As if most people either: 1) Don’t write tests (understandable with Meteor being mostly used for prototyping), 2) Figured everything out but don’t have the time to share (I totally understand that), or 3) Are as lost as I am and live with it but power through with terrible tests 4) Are as lost as I am and don’t write many tests

My learnings

I do have a few things to share that I learned (I wish there was more):

Don’t use hwillson:stub-collections. I was all in at first, because it’s recommended everywhere and I was naive. It speeds up your tests, but my 500 tests run in 7 seconds, so I really don’t need it. And most importantly it mocks your collections, all the packages that modify them (like grapher and simpl-schema/collection2, open issue here), which makes you write tests that don’t actually test your real world app. I think this is a very bad pattern that you should avoid as much as possible, except if you really know what you’re doing. (hint: I don’t)
Put all your meteor methods code in a separate class which is itself called in meteor methods. This was recommended by @diaconutheodor in his great meteor-tuts.com. Meteor methods should only make a security check and then call another class/Service.
Mocking meteor packages: I use babels module-resolver plugin, and then assign meteor to a folder I called meteorStubs, where there is a file for each package:
- src/testHelpers/meteorStubs/
  - meteor.js
  - check.js
  - accounts-base.js
  - cultofcoders:grapher.js
  - etc.
- One issue I’ve encountered, is that Windows based developers can’t use this because “:” is disallowed in their file system. Luckily we’re all on Mac OS
We’ve successfully started using Cypress.io for our end to end tests. It wasn’t completely easy, but we now have a pretty nice setup. An example of which you can see here
Talking about .meteor/packages order, make sure to place aldeed:collection2-core before dburles:factory. Or else be prepared to have violent sweaty nightmares about weirdly failing tests
Use WallabyJS if you can afford it, and use my config as a starting point if it helps. Find it here. This config was actually improved by Artem Govorov from the Wallaby team. He had to dig inside the meteor internals to figure out which babel plugins meteor was secretly using.

What’s next?

I really do not feel confident in my meteor testing skills to push some of this forwards. I wish more was shared and written about really solid testing practices with meteor by senior devs/engineers, that have a good developer experience and can push more people to write more tests.

I’m pretty sure that all the issues I have are hidden somewhere in my code, which is a result of jumping from multiple driver packages, seeing best practices change, and meteor evolving and improving, etc.

I failed to help the community as well by not digging deeper and trying to create reproductions of my issues, but after spending days fighting with an issue and putting my own business in trouble repeatedly, I really can’t justify spending another several hours to extract the weird issues I’m encountering. I just don’t have the ressources. I really wish I did and I could help more. For this I’m sorry, as that makes this post a bit weaker.

These issues really make me regret going with Meteor every now and then, because I’m now very confident that the initial speed boost I got when starting has been easily cancelled out by these endless days figuring out why the meteor magic does what it does (or doesn’t). While the rest of the developer community just moves forwards with widely supported, reliable, well-documented tools, better developer experiences and overall just getting shit done. Spending days figuring out how to write a few meteor tests is preventing me from shipping things.

What I secretly wish for

Here’s a couple things that would make me sleep better:

Near instant test reloads (and regular app reloads as well for that matter)
Proper production-ready testing documentation with all the tricks and tips to write great, scaleable tests for all sorts of use-cases. Given the size of the Testing section (which I will say has improved), testing just really feels like a total afterthought with Meteor.
Snapshot tests ?
Visual regression tests ?
A bigger budget at Qualia to write more awesome advanced meteor tutorials @veered

Please let me know if any of you is also experiencing this frustration, or if I’m alone with this

/rant off

rjdavid · July 20, 2018, 11:21am

My team of one (1) will start to grow in a couple of months. I’m now looking at integrating tests to our code to ensure quality. Thanks for posting this as I’m currently lost and don’t know where to start.

florianbienefelt · July 20, 2018, 3:58pm

Here’s another funny bug I got on circleCI today:

I20180720-13:57:51.831(0)?   1) LoanSecurity isAllowedToUpdate should not do anything if the user is an admin:
I20180720-13:57:51.832(0)?      AssertionError: expected [Function] to not throw an error but 'TypeError: Cannot read property \\'userId\\' of undefined' was thrown
I20180720-13:57:51.832(0)?       at Context.it (app/app.js?hash=cb024010e40dc0748be893f82f5e97a86865ff07:33069:72)
I20180720-13:57:51.832(0)?       at callFn (packages/practicalmeteor_mocha-core.js?hash=fefbc55d41a3a1c9508f13c89ae812d177dac66b:4354:21)
I20180720-13:57:51.832(0)?       at Test.Runnable.run (packages/practicalmeteor_mocha-core.js?hash=fefbc55d41a3a1c9508f13c89ae812d177dac66b:4347:7)
I20180720-13:57:51.832(0)?       at Runner.runTest (packages/practicalmeteor_mocha-core.js?hash=fefbc55d41a3a1c9508f13c89ae812d177dac66b:4817:10)
I20180720-13:57:51.832(0)?       at packages/practicalmeteor_mocha-core.js?hash=fefbc55d41a3a1c9508f13c89ae812d177dac66b:4928:12
I20180720-13:57:51.832(0)?       at next (packages/practicalmeteor_mocha-core.js?hash=fefbc55d41a3a1c9508f13c89ae812d177dac66b:4737:14)
I20180720-13:57:51.833(0)?       at packages/practicalmeteor_mocha-core.js?hash=fefbc55d41a3a1c9508f13c89ae812d177dac66b:4747:7
I20180720-13:57:51.833(0)?       at next (packages/practicalmeteor_mocha-core.js?hash=fefbc55d41a3a1c9508f13c89ae812d177dac66b:4679:14)
I20180720-13:57:51.833(0)?       at packages/practicalmeteor_mocha-core.js?hash=fefbc55d41a3a1c9508f13c89ae812d177dac66b:4715:5
I20180720-13:57:51.833(0)?       at timeslice (packages/practicalmeteor_mocha-core.js?hash=fefbc55d41a3a1c9508f13c89ae812d177dac66b:12683:27)

As you can see the stack trace is complete garbage, how am I supposed to debug a more complicated test?

veered · July 20, 2018, 6:43pm

Anything in particular you’d like a tutorial about?

copleykj · July 21, 2018, 3:31am

I paid almost $40 I believe at one point for “The Meteor Testing Manual” which was never finished and not very complete in the content that it did have. I had high hopes for when it was completed, only to have it be completely abandoned.

dkpconsultingllc · July 22, 2018, 1:04pm

Meteor is great for learning, small projects and newbies. Once one understands every on a much higher level, one can start to see numerous problems and headaches. Even finding meteor packages is a pain. Another being testing. Another being performance via minimongo, the blackbox that is slow slow slow. Limitations on aggregates. Stuck on old mongo. If you go the route of meteor + angular, you are stuck on old angular. Angular 6 (new rxjs) doesnt work with Meteor. Have to use zone.run around all meteor calls since its not built well. Connection issues and resending things twice/three times.

Do what I did, rebuild the server using express and use the oplog and create your own backend. You are then not held back by meteor, can use the latest angular and mongo, can make subs that don’t have to return cursors. 3x performance and stability. Its a no brainer.

awatson1978 · July 22, 2018, 4:06pm

And yet, others of us have been solidly using Nightwatch with Meteor since v0.5, and have successfully migrated the same codebase from PHP to Meteor; and then Backbone to Spark to Blaze to React. We’re successfully running integration tests across an entire release distro of custom packages with meteor test; and bringing Meteor apps into production in mission-critical environments like hospitals. Funny that.

dispatch:mocha is the driver you want.

awatson1978 · July 22, 2018, 4:13pm

florianbienefelt · July 23, 2018, 10:11am

It appears that dispatch:mocha is no longer maintained, and has been renamed to the recommended meteortesting:mocha, is there a reason why you did not upgrade?

Do any of the issues I raised in my post resonate with you, or do you not encounter problems with them at all? I’d be very interested in knowing what your shortlist of improvements would be when building your Meteor testing setup? What painpoints do you encounter at your level?

florianbienefelt · July 23, 2018, 10:30am

@veered Off the top of my head, here are a couple of blog titles I’d click on in a second:

How to identify meteor performance issues with qualia:profiler
- Maybe a few examples of nasty bugs that you can resolve through good use of the profiler. Not sure if this is “simply” a tutorial about how to get better at using the google chrome profiler, which it might be!
Debugging meteor apps
- When you get into really weird issues, what’s your process, how do you set your tools up so that the sourcemaps are good, breakpoints toggle properly, and nothing crashes (all of which I just keep struggling with.)
- I used this VSCode setup, but node crashes, breakpoints don’t always work unverified breakpoint issue. As I’m not sure if you even use VSCode, not sure you’d want to have a too specialized setup to demo to keep it open for all
Advanced Meteor patterns
- Share some cool patterns you built inside Qualia, how you setup your collections, hooks, methods and permissions
- Reading these suggestions from MDG engineers always makes me wonder what’s possible with some DDP-hacking. I think there’s plenty of cool, undocumented methods that would be interesting to learn about
To follow-up on the previous one: “Hacking Meteor: how to make meteor do things you didn’t know it could do”
- Any weird tips and tricks that can make meteor do weird/awesome things. With all the awesome packages that you built, I’m sure you might have a couple of these!
How we built Reval
- Maybe more of a story of how it came to be, what was hard, what you learned
Meteor devOps, how to deploy and maintain meteor at scale
- As you explained in your talks, you seem to have a very interesting setup for deploying your many copies of meteor across the country for each of your customers, sharing some insights into your setup would be amazing.

These are all fairly standard, but maybe it gives you some inspiration !

awatson1978 · July 23, 2018, 1:30pm

Well. Since you mention being a young engineer, I would suggest that you may have a young engineer’s perspective on the value of library ‘maintenance’. Frequent bugfixes and an active issue log aren’t necessarily a sign of the level of completeness, maturity, or stability of a package. Quite the opposite. The most stable ones tend to have accomplished their objectives, and go dormant. What you’re actually optimizing for when you look for libraries ‘that are maintained’ is actually free labor and the latest shiny features. If you want more stable testing, look for libraries that have high trailing semantic versions. I’d trust version 1.3.82 of a package over version 2.0.1 of that same library any day. The former communicates that they’ve been through 82 versions without breaking changes; the later says they’ve just changed/broken the library and it has no history.

Not so much. Your issues are reminiscent of someone who has trained their entire career for sprint racing getting frustrated at getting blisters while running a marathon. You’re pacing yourself wrong and setting up the wrong expectations (in my opinion). The immediacy that you desire in having tests run instantaneously is conspiring against you. Good fast cheap. Those immediate test results are forcing you into Fast/Cheap. Rethinking strategies around tooling and QA processes so testing runs in the background will solve most of the issues.

We’re usually concerned with improving the API to make it isomorphic in new areas; mobile device farms; configuring massive parallelization (hello Walmart’s contribution to the open-source testing world, TestArmada); test code reuse; parameterized testing; etc. That and writing great tests to cover long-term business use cases. And managing code refractors as we replace/upgrade components of our app.

florianbienefelt · July 23, 2018, 3:00pm

@awatson1978 Yes, you’re probably right. I’m well aware of the biased way in which I currently choose the libraries I use, even Meteor itself. I think the JS community as a whole is probably stumbling on this, except for more experienced engineers like you, I hope.

Given the amount of package decisions we all have to make on a regular basis, I think it’s unreasonable to expect any (young) solo-developer to thoroughly vet every decision by inspecting the code, or checking whether a package works properly. I have to maintain a certain level of pragmatism. In regards to optimizing for free labor, of course I optimize in part for that, and in the open-source world, what kind of unreasonable person would not?
Looking at activity and popularity of libraries/frameworks is to me a acceptable (though obviously not perfect) proxy to quickly make decisions, at the risk of getting hurt every now and then (which is ok).

And yes, while I do this for a living, I’m passionate about technology, and new tools/features, so I do upgrade and switch faster than I should, but it keeps me entertained and happy to grind. I’ll pull a couple more days a month to fix the issues I put myself in because of this habit. Maybe you’re (understandably so) more conservative and risk-averse as you’ve been through this already. I guess I still need a few hard burns until I start thinking like that as well.

Now to your comment about complaining about blisters… I find it unnecessary. I consider myself able to pick up things reasonable fast, and I prefaced this rant by saying that I have been building meteor apps for roughly 2 years and that I still fall into really basic traps when writing meteor tests. I just can’t see much improvement in my meteor-testing skills, which is clearly an outlier to me, hence the introspection and the essence of this post.

Regarding speed: I can see this with myself and the younger engineers I’m starting to work with, to get one relatively complicated test to pass, they have to tweak it 10-30x, so if you want to build up their skill and confidence fast, having them wait 10 seconds every time they hit save it not the way to do it. I fail to understand why you would Not want that to improve, there’s literally no reason to prefer the status-quo (except for MDG developers not spending time on what you believe is more important, granted).

Regarding your Venn diagram, I think any conversation about tests is already pretty far from “fast and cheap”. I’m concerned about making “writing good Meteor software” a better experience for myself and other younger devs, and in that area, I believe there’s a lot to improve. Just look at the amount of testing issues on this forum, there are barely any (300 in that category since the beginning of this forum). Clearly, too few people are concerned with it.

Thanks for the candid feedback.

alawi · July 23, 2018, 3:14pm

Good, educational, mature and candid conversation, nice read. A lot of times it’s more about the community then the tools.

veered · July 23, 2018, 5:38pm

Thanks for the ideas! It definitely gives me some inspiration. I love writing up or building tools about this sort of stuff

veered · July 23, 2018, 5:43pm

One trick that I always thought was pretty cool is this technique for telling whether or not you are in a Meteor method called by another Meteor method:

let Fiber = Npm.require('fibers');
function isInternalCall() {
  // Dark, dark magic...
  if (Fiber.current._meteor_dynamics[0]) {
    return Fiber.current._meteor_dynamics[0].description.startsWith('internal');
  }
  return false;
}

This can be a helpful tool for building out better logging or permissioning (nested Meteor method calls might have greater privileges).

Qualia uses a lot of weird tricks like this one

hexsprite · July 23, 2018, 6:21pm

I usually just check this.connection to determine if it was called via DDP or internally

serkandurusoy · July 25, 2018, 9:04am

This topic sounds like it is becoming one of the most rewarding reads on this forum. Thanks everyone for contributing.

@florianbienefelt can you remember any specific problems you’ve had with meteortesting:mocha? Maybe if we broke down the problems and tackled them individually, we can identify the most important bottlenecks and make progress from there?

rooch84 · July 27, 2018, 8:32am

Just wanted to throw my 2p in - I’ve hit many of the same issues you describe. Component tests were a non-starter. I tried Chimp for a while, but got very different behaviours with the headed and headless versions. I now only run end-to-end tests with Cypress (which is a lovely tool).

@dkpconsultingllc and @awatson1978 - it’s interesting to see two very contrasting views. I’ve been toying with dropping Meteor recently. We moved from a GWT prototype to a Meteor + React implementation with the aim of creating something more commercially viable. Even though development is much much faster, what we have now in Meteor actually appears to run significantly slower than the GWT prototype (although, this could be the React side of the implementation). I now need to figure out if it is Meteor that is slower, or my lack of understanding of how things behave under the hood.

alawi · July 27, 2018, 11:49am

Run slower? where in production? I used GWT for years and built two commercial apps, I really don’t understand how react/meteor run slower, GWT was very bloated. Not sure how and what you measured to come to this conclusion.

stolinski · July 27, 2018, 2:51pm

Really nice post. I’ve been frustrated with the state of testing myself. IMO just more solid best practices up to date public code would make things easier to get up with.