Testing Meteor apps in 2018, a rant

[stolinski]

I’m interested in hearing how others are testing their serverside Meteor code specifically. Since I’m using Jest / React Testing Library / Apollo, client side testing has been a breeze, but I’ve been having some issues testing serverside. Specifically integrating Jest into Meteor.

I use https://github.com/orangecms/jest-meteor-stubs but I’d prefer something that integrated better than just stubbing Meteor packages straight up. For instance, testing things like having mutations actually insert correctly into Mongo with things like SimpleSchema checking.

If anyone has an good repos I’d love to see them.

[namirsab]

Hi there. Interesting topic for sure.
I’m having issues as well when trying to test packages. I’ve had issues since I started working with Meteor (3 years ago, when I used to use velocity + jasmine) and I always fixed them. We moved to spacejam afterwards, and everything was working fine. we are using the meteortesting:mocha driver package, which was working properly, but now i get this error:

W20180820-18:37:48.586(2)? (STDERR) The version of @babel/runtime installed in your node_modules directory 
W20180820-18:37:48.587(2)? (STDERR) (7.0.0-rc.1) contains a breaking change which was introduced by 
W20180820-18:37:48.587(2)? (STDERR) https://github.com/babel/babel/pull/8266. Please either downgrade by 
W20180820-18:37:48.587(2)? (STDERR) running the following command:
W20180820-18:37:48.588(2)? (STDERR) 
W20180820-18:37:48.588(2)? (STDERR)   meteor npm install --save-exact @babel/runtime@7.0.0-beta.55
W20180820-18:37:48.588(2)? (STDERR) 
W20180820-18:37:48.588(2)? (STDERR) or update to the latest beta version of Meteor 1.7.1, as explained in 
W20180820-18:37:48.588(2)? (STDERR) this pull request: https://github.com/meteor/meteor/pull/9942.
W20180820-18:37:48.589(2)? (STDERR) 
W20180820-18:37:48.589(2)? (STDERR) /tmp/meteor-test-runu20rfu.zo4jg/.meteor/local/build/programs/server/boot.js:475
W20180820-18:37:48.589(2)? (STDERR) }).run();
W20180820-18:37:48.589(2)? (STDERR)    ^
W20180820-18:37:48.589(2)? (STDERR) 
W20180820-18:37:48.590(2)? (STDERR) Error: Cannot find module './defineProperty'
W20180820-18:37:48.590(2)? (STDERR)     at makeMissingError (packages/modules-runtime.js:231:12)
W20180820-18:37:48.590(2)? (STDERR)     at require (packages/modules-runtime.js:241:19)
W20180820-18:37:48.590(2)? (STDERR)     at objectSpread.js (packages/modules.js:513:22)
W20180820-18:37:48.590(2)? (STDERR)     at fileEvaluate (packages/modules-runtime.js:343:9)
W20180820-18:37:48.591(2)? (STDERR)     at require (packages/modules-runtime.js:238:16)
W20180820-18:37:48.591(2)? (STDERR)     at logging.js (/tmp/meteor-test-runu20rfu.zo4jg/.meteor/local/build/programs/server/packages/logging.js:26:45)
W20180820-18:37:48.591(2)? (STDERR)     at fileEvaluate (packages/modules-runtime.js:343:9)
W20180820-18:37:48.591(2)? (STDERR)     at require (packages/modules-runtime.js:238:16)
W20180820-18:37:48.591(2)? (STDERR)     at /tmp/meteor-test-runu20rfu.zo4jg/.meteor/local/build/programs/server/packages/logging.js:511:15
W20180820-18:37:48.592(2)? (STDERR)     at /tmp/meteor-test-runu20rfu.zo4jg/.meteor/local/build/programs/server/packages/logging.js:518:3

I’ve tried everything, but it just does not work.

That said, I think the biggest problem regarding Meteor testing is the lack of continuity. Every new releaese tends to break the existing way of testing. That’s frustrating…

Is anyone also having this issue?

[cmoses8626]

Thanks for the good discussion. I’m interested in learning - how do you structure your tests? Given this discussion, are you importing your *.test.* files into /tests/main.js so they’re properly loaded, putting them into /tests sub-folders (e.g. /tests/api), or putting all your tests into /tests/main.js?

[awatson1978]

Actually, we do something way more wonky: we keep verification and integration tests in sub-folders within packages, and copy them to the /imports directory as a step during continuous-integration.

clinical-meteor/hl7-resource-observation/blob/master/.circleci/config.yml#L102-L127

We also keep validation/acceptance tests in /tests/nightwatch.

[hexsprite]

I use a naming scheme to keep all my tests organized.

*.app-spec.js for all my integration tests with Meteor (meteortesting:mocha)
*.unit-spec.js for unit tests using Jest.

These are all co-located with the code that is being tested to make it easy to maintain.

I have /tests/e2e using chimp.

[drollins]

I use cultofcoders:mocha in 1.7.0.4 with no issue

[stolinski]

Has anyone been using Jest to test server side? Specifically with Meteor packages. I’d prefer not to have to use Mocha and Jest in the repo and using Mocha for everything seems like a huge bummer.

[pjtavares]

Great initial post (echos some of my frustration as well) and ongoing thread. and, I agree, it would certainly be helpful for someone from MDG to comment here as well!

I’m with stolinski, I’d love to see a working test driver option for Jest. I’m willing to take some action to help but, having only worked with Meteor for just over a year, some guidance would be helpful. As far as I can tell, there is no documentation for writing test drivers. I suppose I could try and piece it together from the meteortesting:mocha or cultofcoders:mocha source code but I’d like to speed up that learning time if possible. Is anyone aware of anything helpful here?

[stolinski]

Likewise. I’d be down to hack together something with you. My needs are really just to be able to write tests and have them run in the command line. I don’t need browser outputting or anything like that. Might make the whole thing a bit easier.

[rjdavid]

We’ve successfully started using Cypress.io for our end to end tests. It wasn’t completely easy, but we now have a pretty nice setup.

Was able to setup Cypress for our e2e regression tests. This is just an amazing testing tool. It can run both with gui and headless. The headless version was perfect for CI.

[florianbienefelt]

Took the time to create an issue for my completely unreasonable circle CI app build times (>20 mins): https://github.com/meteor/meteor/issues/10218

Hoping nobody has to debug a 20 minute long process, it’s pure horror

[awatson1978]

We’ve been running into that issue of late also. We’ve managed to keep the build time under 5 minutes, but there’s still a 10 minute timeout and it’s usually 8 minutes before tests are running.

Are you using caching? We had caching turned on in circle 1.0 but haven’t gotten it up and working yet on 2.0. Would be great to compare notes on the circleci/config.yml file.

Also, we recently discovered that one of our heisenbugs in the new build was related to an external dependency. We were fetching an online library that had ~50MB+ of data as part of the application, which was causing big problems. The new Circle CI containers seem to have slower network connections or are throttled compared to 1.0. Putting that feature behind an environment variable dependent conditional got the tests running smoothly again.

[awatson1978]

In theory, the following syntax should help get caching working, but I haven’t gotten the pattern working 100% yet…

    # Dependencies
    #   This would typically go in either a build or a build-and-test job when using workflows
    # # Restore the dependency cache
    # - restore_cache:
    #     keys:
    #     # This branch if available
    #     - v1-dep-{{ .Branch }}-
    #     # Default branch if not
    #     - v1-dep-development-
    #     # Any branch if there are none on the default branch - this should be unnecessary if you have your default branch configured correctly
    #     - v1-dep-
        
    # # Save dependency cache
    # - save_cache:
    #     name: Saving cache
    #     key: v1-dep-{{ .Branch }}-{{ epoch }}
    #     paths:
    #     # This is a broad list of cache paths to include many possible development environments
    #     # You can probably delete some of these entries
    #     - ~/.meteor

[tomsp]

@awatson1978 I recently went through debugging our ci config (20 mins per run, what a nightmare) and here is what we use for caching now.

# Restore dependency cache section:
- restore_cache:
    name: Restore Meteor bin cache
    key: meteor-bin-cache-{{ checksum ".meteor/release" }}
- restore_cache:
    name: Restore Meteor cache
    key: meteor-cache-{{ checksum ".meteor/release" }}
- restore_cache:
    name : Restore NPM/Yarn cache
    key: npm-cache-{{ checksum "package.json" }} # should use package.lock/yarn.lock here
- restore_cache:
    name: Restore Meteor Package cache
    key: packages-cache-{{ checksum ".meteor/versions" }}

# Install dependency cache section:
- run:
    name: Restore or install Meteor
    command: |
      if [ -e ~/build-temp/meteor-bin ]
      then
          echo "Cached Meteor bin found, restoring it"
          sudo cp ~/build-temp/meteor-bin /usr/local/bin/meteor
      else
          echo "No cached Meteor bin found."
          curl https://install.meteor.com | /bin/sh
      fi
- run: meteor --version
- run: meteor list # you might think you can remove this line but if we do our builds time out almost every time.
- run: meteor npm install -g yarn
- run: meteor yarn

# Save dependency cache section:
- save_cache:
    name: Save Meteor cache
    key: meteor-cache-{{ checksum ".meteor/release" }}
    paths:
      - '~/.meteor'
- run:
    name: Copy Meteor bin to build cache
    command: |
      mkdir -p ~/build-temp
      cp /usr/local/bin/meteor ~/build-temp/meteor-bin
- save_cache:
    name: Save Meteor bin cache
    key: meteor-bin-cache-{{ checksum ".meteor/release" }}
    paths:
      - '~/build-temp'
- save_cache:
    name: Save NPM/Yarn cache
    key: npm-cache-{{ checksum "package.json" }}
    paths:
      - 'node_modules'
- save_cache:
    name: Save Meteor Package cache
    key: packages-cache-{{ checksum ".meteor/versions" }}
    paths:
      - './.meteor/local/build'
      - './.meteor/local/bundler-cache'
      - './.meteor/local/isopacks'
      - './.meteor/local/plugin-cache'

[awatson1978]

Oh, that’s fantastic @tomsp! Thank you SOOOO much! Christmas in September!

[rooch84]

Having spent the summer refactoring our codebase, and will a demo due next week, now feels like a good time to reply.

I have to say, I wholeheartedly retract my post above. Most of the issues we experienced were related to either poor code or a lack of understanding how minimongo was working under the hood. What we have now is something that runs really fast. We’ll be sticking with Meteor + React for the foreseeable future. Our next refactor may be to switch from a back-end REST end-point to GraphQL.

Cheers
Chris

[hemalr87]

Came on here to search for a testing best-practices guide and found this (which reflects a lot of the test-related search results I’ve gotten for Meteor).

I, embarrassingly, have very little automated testing set up on my production app, but that obviously needs to change. The massive checklist of manual testing I’ve created for myself works, but is time consuming (feel free to shudder, no comments necessary) and that goes up every time I want to update anything.

I wonder whether, as a community, we can agree on a set of testing tools and work together to write a Gitbook or similar with an opinionated guide on testing Meteor. Would anyone else be interested?

Tools in my mind (open to suggestion as I have little experience with it):

1. Jest
2. React-Testing-Library
3. Cypress

(Tools picked purely because that’s what Kent Dodds used in his testing course…)

[mattblackdev]

I’m willing to help a jest test driver. Sounds like the way to go. cc/@stolinski @pjtavares

[jkuester]

With react you save lots of time if you create many small generic and reusable stateless components. For those you won’t need complex tests with webdriver/chimp etc. but can write static granular unit tests. Saves lots of time in the long run.

[jam]

Curious if you or others made any headway with a Jest driver?