Testing Meteor apps in 2018, a rant

[mhinton]

We have also be frustrated with the state of testing and the documentation provided recently. We used to use the browser reporter that ran client and server tests and were quite happy with it. We upgraded a Meteor app and had to change the testing setup again and we have now lost the browser based test reporter. Currently we are using phantomjs and editing files doesn’t seem to rerun tests using this npm script.

"test": "TEST_WATCH=1 TEST_BROWSER_DRIVER=phantomjs meteor test --driver-package meteortesting:mocha --port 3100 --settings ./private/settings.json"

Maybe I am using the wrong option of the many listed in the testing guide, but again the guide is not very clear. If the testing guide marked which sections refer to previous versions of Meteor that would be helpful. It would be nice if there were example projects for different testing setups. At the least I think that meteor create --full should have more test scripts that illustrate server / client / headless test cases.

[jpperrin]

I’ve just started building tests into my meteor application, and am not having issues yet.

I’ve been using
cultofcoders:mocha

Works great in the browser, and does the trick for me.

Granted I’ve only written server tests so far, but i’ll be keeping an eye on this thread.

[rhywden]

Yeah, but that one blows up with 1.7.1-beta. @diaconutheodor

This is from a “virgin” project where nothing much has been done yet and only includes the standard tests.

And the “official” testing package reference in the Meteor guide just plain crashes right from the start.

[rijk]

Really enjoying this topic. Thanks for all the constructive feedback/criticism. Testing in Meteor definitely seems to need some TLC. It would be great if someone from the MDG could weigh in with their 2 cents (@sashko? @benjamn?).

[stolinski]

I’m interested in hearing how others are testing their serverside Meteor code specifically. Since I’m using Jest / React Testing Library / Apollo, client side testing has been a breeze, but I’ve been having some issues testing serverside. Specifically integrating Jest into Meteor.

I use https://github.com/orangecms/jest-meteor-stubs but I’d prefer something that integrated better than just stubbing Meteor packages straight up. For instance, testing things like having mutations actually insert correctly into Mongo with things like SimpleSchema checking.

If anyone has an good repos I’d love to see them.

[namirsab]

Hi there. Interesting topic for sure.
I’m having issues as well when trying to test packages. I’ve had issues since I started working with Meteor (3 years ago, when I used to use velocity + jasmine) and I always fixed them. We moved to spacejam afterwards, and everything was working fine. we are using the meteortesting:mocha driver package, which was working properly, but now i get this error:

W20180820-18:37:48.586(2)? (STDERR) The version of @babel/runtime installed in your node_modules directory 
W20180820-18:37:48.587(2)? (STDERR) (7.0.0-rc.1) contains a breaking change which was introduced by 
W20180820-18:37:48.587(2)? (STDERR) https://github.com/babel/babel/pull/8266. Please either downgrade by 
W20180820-18:37:48.587(2)? (STDERR) running the following command:
W20180820-18:37:48.588(2)? (STDERR) 
W20180820-18:37:48.588(2)? (STDERR)   meteor npm install --save-exact @babel/runtime@7.0.0-beta.55
W20180820-18:37:48.588(2)? (STDERR) 
W20180820-18:37:48.588(2)? (STDERR) or update to the latest beta version of Meteor 1.7.1, as explained in 
W20180820-18:37:48.588(2)? (STDERR) this pull request: https://github.com/meteor/meteor/pull/9942.
W20180820-18:37:48.589(2)? (STDERR) 
W20180820-18:37:48.589(2)? (STDERR) /tmp/meteor-test-runu20rfu.zo4jg/.meteor/local/build/programs/server/boot.js:475
W20180820-18:37:48.589(2)? (STDERR) }).run();
W20180820-18:37:48.589(2)? (STDERR)    ^
W20180820-18:37:48.589(2)? (STDERR) 
W20180820-18:37:48.590(2)? (STDERR) Error: Cannot find module './defineProperty'
W20180820-18:37:48.590(2)? (STDERR)     at makeMissingError (packages/modules-runtime.js:231:12)
W20180820-18:37:48.590(2)? (STDERR)     at require (packages/modules-runtime.js:241:19)
W20180820-18:37:48.590(2)? (STDERR)     at objectSpread.js (packages/modules.js:513:22)
W20180820-18:37:48.590(2)? (STDERR)     at fileEvaluate (packages/modules-runtime.js:343:9)
W20180820-18:37:48.591(2)? (STDERR)     at require (packages/modules-runtime.js:238:16)
W20180820-18:37:48.591(2)? (STDERR)     at logging.js (/tmp/meteor-test-runu20rfu.zo4jg/.meteor/local/build/programs/server/packages/logging.js:26:45)
W20180820-18:37:48.591(2)? (STDERR)     at fileEvaluate (packages/modules-runtime.js:343:9)
W20180820-18:37:48.591(2)? (STDERR)     at require (packages/modules-runtime.js:238:16)
W20180820-18:37:48.591(2)? (STDERR)     at /tmp/meteor-test-runu20rfu.zo4jg/.meteor/local/build/programs/server/packages/logging.js:511:15
W20180820-18:37:48.592(2)? (STDERR)     at /tmp/meteor-test-runu20rfu.zo4jg/.meteor/local/build/programs/server/packages/logging.js:518:3

I’ve tried everything, but it just does not work.

That said, I think the biggest problem regarding Meteor testing is the lack of continuity. Every new releaese tends to break the existing way of testing. That’s frustrating…

Is anyone also having this issue?

[cmoses8626]

Thanks for the good discussion. I’m interested in learning - how do you structure your tests? Given this discussion, are you importing your *.test.* files into /tests/main.js so they’re properly loaded, putting them into /tests sub-folders (e.g. /tests/api), or putting all your tests into /tests/main.js?

[awatson1978]

Actually, we do something way more wonky: we keep verification and integration tests in sub-folders within packages, and copy them to the /imports directory as a step during continuous-integration.

clinical-meteor/hl7-resource-observation/blob/master/.circleci/config.yml#L102-L127

We also keep validation/acceptance tests in /tests/nightwatch.

[hexsprite]

I use a naming scheme to keep all my tests organized.

*.app-spec.js for all my integration tests with Meteor (meteortesting:mocha)
*.unit-spec.js for unit tests using Jest.

These are all co-located with the code that is being tested to make it easy to maintain.

I have /tests/e2e using chimp.

[drollins]

I use cultofcoders:mocha in 1.7.0.4 with no issue

[stolinski]

Has anyone been using Jest to test server side? Specifically with Meteor packages. I’d prefer not to have to use Mocha and Jest in the repo and using Mocha for everything seems like a huge bummer.

[pjtavares]

Great initial post (echos some of my frustration as well) and ongoing thread. and, I agree, it would certainly be helpful for someone from MDG to comment here as well!

I’m with stolinski, I’d love to see a working test driver option for Jest. I’m willing to take some action to help but, having only worked with Meteor for just over a year, some guidance would be helpful. As far as I can tell, there is no documentation for writing test drivers. I suppose I could try and piece it together from the meteortesting:mocha or cultofcoders:mocha source code but I’d like to speed up that learning time if possible. Is anyone aware of anything helpful here?

[stolinski]

Likewise. I’d be down to hack together something with you. My needs are really just to be able to write tests and have them run in the command line. I don’t need browser outputting or anything like that. Might make the whole thing a bit easier.

[rjdavid]

We’ve successfully started using Cypress.io for our end to end tests. It wasn’t completely easy, but we now have a pretty nice setup.

Was able to setup Cypress for our e2e regression tests. This is just an amazing testing tool. It can run both with gui and headless. The headless version was perfect for CI.

[florianbienefelt]

Took the time to create an issue for my completely unreasonable circle CI app build times (>20 mins): https://github.com/meteor/meteor/issues/10218

Hoping nobody has to debug a 20 minute long process, it’s pure horror

[awatson1978]

We’ve been running into that issue of late also. We’ve managed to keep the build time under 5 minutes, but there’s still a 10 minute timeout and it’s usually 8 minutes before tests are running.

Are you using caching? We had caching turned on in circle 1.0 but haven’t gotten it up and working yet on 2.0. Would be great to compare notes on the circleci/config.yml file.

Also, we recently discovered that one of our heisenbugs in the new build was related to an external dependency. We were fetching an online library that had ~50MB+ of data as part of the application, which was causing big problems. The new Circle CI containers seem to have slower network connections or are throttled compared to 1.0. Putting that feature behind an environment variable dependent conditional got the tests running smoothly again.

[awatson1978]

In theory, the following syntax should help get caching working, but I haven’t gotten the pattern working 100% yet…

    # Dependencies
    #   This would typically go in either a build or a build-and-test job when using workflows
    # # Restore the dependency cache
    # - restore_cache:
    #     keys:
    #     # This branch if available
    #     - v1-dep-{{ .Branch }}-
    #     # Default branch if not
    #     - v1-dep-development-
    #     # Any branch if there are none on the default branch - this should be unnecessary if you have your default branch configured correctly
    #     - v1-dep-
        
    # # Save dependency cache
    # - save_cache:
    #     name: Saving cache
    #     key: v1-dep-{{ .Branch }}-{{ epoch }}
    #     paths:
    #     # This is a broad list of cache paths to include many possible development environments
    #     # You can probably delete some of these entries
    #     - ~/.meteor

[tomsp]

@awatson1978 I recently went through debugging our ci config (20 mins per run, what a nightmare) and here is what we use for caching now.

# Restore dependency cache section:
- restore_cache:
    name: Restore Meteor bin cache
    key: meteor-bin-cache-{{ checksum ".meteor/release" }}
- restore_cache:
    name: Restore Meteor cache
    key: meteor-cache-{{ checksum ".meteor/release" }}
- restore_cache:
    name : Restore NPM/Yarn cache
    key: npm-cache-{{ checksum "package.json" }} # should use package.lock/yarn.lock here
- restore_cache:
    name: Restore Meteor Package cache
    key: packages-cache-{{ checksum ".meteor/versions" }}

# Install dependency cache section:
- run:
    name: Restore or install Meteor
    command: |
      if [ -e ~/build-temp/meteor-bin ]
      then
          echo "Cached Meteor bin found, restoring it"
          sudo cp ~/build-temp/meteor-bin /usr/local/bin/meteor
      else
          echo "No cached Meteor bin found."
          curl https://install.meteor.com | /bin/sh
      fi
- run: meteor --version
- run: meteor list # you might think you can remove this line but if we do our builds time out almost every time.
- run: meteor npm install -g yarn
- run: meteor yarn

# Save dependency cache section:
- save_cache:
    name: Save Meteor cache
    key: meteor-cache-{{ checksum ".meteor/release" }}
    paths:
      - '~/.meteor'
- run:
    name: Copy Meteor bin to build cache
    command: |
      mkdir -p ~/build-temp
      cp /usr/local/bin/meteor ~/build-temp/meteor-bin
- save_cache:
    name: Save Meteor bin cache
    key: meteor-bin-cache-{{ checksum ".meteor/release" }}
    paths:
      - '~/build-temp'
- save_cache:
    name: Save NPM/Yarn cache
    key: npm-cache-{{ checksum "package.json" }}
    paths:
      - 'node_modules'
- save_cache:
    name: Save Meteor Package cache
    key: packages-cache-{{ checksum ".meteor/versions" }}
    paths:
      - './.meteor/local/build'
      - './.meteor/local/bundler-cache'
      - './.meteor/local/isopacks'
      - './.meteor/local/plugin-cache'

[awatson1978]

Oh, that’s fantastic @tomsp! Thank you SOOOO much! Christmas in September!

[rooch84]

Having spent the summer refactoring our codebase, and will a demo due next week, now feels like a good time to reply.

I have to say, I wholeheartedly retract my post above. Most of the issues we experienced were related to either poor code or a lack of understanding how minimongo was working under the hood. What we have now is something that runs really fast. We’ll be sticking with Meteor + React for the foreseeable future. Our next refactor may be to switch from a back-end REST end-point to GraphQL.

Cheers
Chris