Meteor-node-stubs: When is it really needed?

help
1

I am running Meteor inside a mobile app and currently trying to get its bundle size as low as possible. When checking it with the bundle-visualized, I realized that meteor-node-stubs adds a significant amount of weight.

So I am wondering when it is actually needed in an app?

I found this old thread (where I even asked the same question years ago), but the answers were a bit cryptic:

According to Simon, it is used…

to allow using npm packages written for node on the browser

But what does that mean in practice? Are there any important Meteor packages that actually require this? How can I know for sure if this is the case?

In the Guide

https://guide.meteor.com/using-npm-packages.html

I found the following statement:

Meteor’s module system avoids actually bundling any stub modules (and their dependencies) if they are not used, so there is no cost to keeping meteor-node-stubs in the dependencies. In other words, leave meteor-node-stubs installed unless you really know what you’re doing.

Does that mean that because it appears in my client-side bundle, it is being used by some other package I am not aware of?

The biggest contribution to package size comes from elliptic, which seem to be a crypto library. Is this used by meteor-accounts somehow? I am not using any crypto stuff besides that.

1 Like
2

Maybe you could try madge in order to find out what exactly is referencing meteor-node-stubs in your app:

Madge is a developer tool for generating a visual graph of your module dependencies, finding circular dependencies, and give you other useful info. Joel Kemp’s awesome dependency-tree is used for extracting the dependency tree.

3 Likes
3

Sounds good, will try it out!

EDIT: Tried it, but couldn’t get any useful information out of it. It creates huge graphs, but couldn’t really answer the question which other code references meteor-node-stubs.

4

I am not sure but since elliptic ist part of crypto-browserify we may be able to replace it since modern browsers implement the web crypto api

1 Like
5

I also wonder if every node package, listed in the mapping file is really required by default :thinking:

Edit: okay now it’s getting interesting. I uninstalled meteor-node-stubs completely and restarted my app and created new accounts (using accounts-password) but it seems that nowhere there is any error thrown. The whole operation now freed me 77 packages.

Edit edit: when running via --production flag then I get the first errors, for example

Uncaught (in promise) Error: Cannot find module ‘assert’

Now I am wondering where and why assert is really needed :sweat_smile:
→ Found → Came from bundle-visualizer but apart from it I found no usage of any of them.

3 Likes
6

Interesting. According to the docs, the meteor stubs would only be bundled if they are actually needed.

7

Okay it seems that I can’t run headless tests without them, the client environment is simply missing things but fails silently. When adding the package it runs all fine. Maybe we can work through them to find a way to only have those packages stubbed that are really needed. I am opening an issue on GitHub since this is not intended behaviour

2 Likes
8

I did some research using bundle-visualizer and Meteor 2.x with the following combinations:

Fresh empty projects

command size packages
meteor create 3.61kb process, assert
meteor create --minmal 3.61kb process, assert
meteor create --bare 3.61kb process, assert
meteor create --full 3.61kb process, assert
meteor create --blaze 3.61kb process, assert
meteor create --typescript 3.61kb process, assert
meteor create --vue 3.61kb process, assert
meteor create --apollo 3.61kb process, assert

As you can see, there is minimal to no usage of any required stubs when creating new projects. Note, that assert is added by bundle-visualizer.

meteor create with extra packages

In order to find the cause for these big stubs we should check for certain popular packages (please comment for packages to include).

extra size-increase packages added
accounts-base - -
accounts-password - -
dynamic-import - -
Random - -
fourseven:scss, seba:minifiers-autoprefixer - -
hot-module-replacement - -
jquery (Meteor, 3.0.0) - -
meteortesting:mocha - -
facts-base, facts-ui - -
email - -
ongoworks:ddp-login - -
force-ssl - -
lmieulet:meteor-coverage - -
hwillson:stub-collections - -

With extra NPM packages

The following npm packages are just a selection of packages I found in some projects.

extra size-increase packages added
simple-schema - -
bcrypt (npm) - -
bootstrap, popper, jquery - -
@fortawesome/fontawesome-free - -
chai (dev) - -
sinon (dev) - -
validator - -
mmmagic - -
sortablejs - -
gridfs-stream 72.99kb too many
gm 77,14kb too many
mime-types 4,29kb path-browserify
crypto 630kb way too many :scream:

Note: the increase means the increase of meteor-node-stubs.

As you can see, the source is mostly in NPM packages, that are targeting node but somehow imported on the client.

I think this is mostly a code-splitting issue, you need to find the packages, that are implicitly imported on the client but are not really used and manage to not get them into the import graph.

Summary

The biggest hidden dependencies that make node stubs grow are somewhere in your package.json dependencies (or packages that use Npm.depends). These are very likely dependencies that target the node platform (and thus use packages like ‘fs’ , ‘utils’, ‘stream’ or ‘crypto’ etc.) but are imported to the client, either intended or unintended or as dependency of an intended import.

7 Likes
9

@waldgeist can you please post your .meteor/packages and your pacakges.json dependencies so I can add them to the list. There must be a package somewhere that causes the increase. Also - do you remember the first Meteor release version your project had when you created it?

10

This is amazing, really interesting. Can you add autoform? I don’t think it will add much as it’s a large but “simple” package.

Can you explain the increase column a little more? Are you saying that adding the crypto package makes the node-stubs package grow by 630kb? Or it increases the overall bundle size by this much?

11

It increased node-stubs but in return the overall size, of course, too. Now it’s getting even crazier - if an NPM package reuiqres a stub plus dependencies then the bundle can easily grow by more than 1MB :open_mouth:

I’m currently writing an article on this topic

4 Likes
12

First off, thanks a lot for this thorough investigation. That’s really interesting!

My initial Meteor version was 1.8.2, currently I’m at 1.10.2.

Here’s my dependencies

packages

meteor-base@1.4.0             # Packages every Meteor app needs to have
mobile-experience@1.1.0       # Packages for a great mobile UX
mongo@1.10.0                   # The database Meteor supports right now
reactive-var@1.0.11            # Reactive variable for tracker

standard-minifier-css@1.6.0   # CSS minifier run for production mode
standard-minifier-js@2.6.0    # JS minifier run for production mode
es5-shim@4.8.0                # ECMAScript 5 compatibility for older browsers
ecmascript@0.14.3              # Enable ECMAScript2015+ syntax in app code
shell-server@0.5.0            # Server-side component of the `meteor shell` command

static-html             # Define static page content in .html files
react-meteor-data@2.1.0       # React higher-order component for reactively tracking Meteor data
aldeed:collection2
aldeed:schema-index
http@1.4.2
session@1.2.0
#swydo:ddp-apollo
accounts-password@1.6.0
audit-argument-checks@1.0.7
edgee:slingshot
meteoreact:accounts
#meteoreact:accounts-unstyled
percolate:migrations
#dispatch:login-token
loren:login-links
fourseven:scss
alanning:roles
littledata:synced-cron
nspangler:autoreconnect
meteortesting:mocha
kadira:dochead
montiapm:agent
reywood:publish-composite
quave:accounts-apple
accounts-facebook
bozhao:link-accounts
accounts-google
meteorhacks:search-source
check
universe:i18n
lamhieu:unblock
johanbrook:publication-collector

packages.json

    "@babel/runtime": "^7.9.6",
    "@sandstreamdev/react-swipeable-list": "^1.0.0",
    "@turf/circle": "^6.0.1",
    "@urbica/react-map-gl": "^1.14.2",
    "@urbica/react-map-gl-cluster": "^0.2.0",
    "@videojs/themes": "^1.0.0",
    "adm-zip": "^0.4.13",
    "aws4": "^1.10.1",
    "aws4-axios": "^1.12.0",
    "axios": "^0.21.1",
    "bcrypt": "^3.0.8",
    "blueimp-canvas-to-blob": "^3.14.0",
    "blueimp-load-image": "^2.21.0",
    "chart.js": "^2.9.3",
    "classnames": "^2.2.6",
    "compare-versions": "^3.6.0",
    "connected-react-router": "^6.3.2",
    "csvjson": "^5.1.0",
    "dashjs": "^3.1.2",
    "data-uri-to-blob": "0.0.4",
    "dayjs": "^1.10.3",
    "diacritics": "^1.3.0",
    "dot-object": "^2.1.3",
    "events": "^3.0.0",
    "exifreader": "^2.8.3",
    "file-saver": "^2.0.2",
    "form-data": "^3.0.0",
    "haversine": "^1.1.1",
    "history": "^4.9.0",
    "i18n-iso-countries": "^4.1.0",
    "indexof": "0.0.1",
    "lodash.debounce": "^4.0.8",
    "lodash.unionby": "^4.8.0",
    "mapbox-gl": "^1.13.1",
    "meteor-node-stubs": "^0.4.1",
    "node-pushnotifications": "^1.5.0",
    "nuka-carousel": "^4.5.12",
    "papaparse": "^5.2.0",
    "prerender-node": "^3.2.5",
    "prop-types": "^15.7.2",
    "query-string": "^6.8.1",
    "randimal": "^1.0.0",
    "rc-slider": "^8.7.1",
    "react": "^16.11.0",
    "react-avatar-edit": "^0.8.3",
    "react-avatar-editor": "^11.0.7",
    "react-bootstrap": "^1.0.0-beta.16",
    "react-bootstrap-typeahead": "^3.4.7",
    "react-dom": "^16.11.0",
    "react-flip-toolkit": "^7.0.7",
    "react-redux": "^6.0.1",
    "react-router-bootstrap": "^0.25.0",
    "react-router-dom": "^5.2.0",
    "react-slick": "^0.25.2",
    "react-swipeable": "^5.5.0",
    "react-switch": "^6.0.0",
    "react-tag-autocomplete": "^6.0.0-beta.3",
    "react-textarea-autosize": "^7.1.0",
    "react-virtualized": "^9.21.1",
    "redux": "^4.0.1",
    "redux-thunk": "^2.3.0",
    "semver-compare": "^1.0.0",
    "simpl-schema": "^1.5.5",
    "slugify": "^1.4.5",
    "supercluster": "^6.0.0",
    "token": "^0.1.0",
    "ua-parser-js": "^0.7.21",
    "uid-generator": "^2.0.0",
    "underscore.string": "^3.3.5",
    "uuid": "^3.3.3",
    "valvelet": "^1.1.1",
    "velocity-react": "^1.4.3",
    "video.js": "^7.11.4",
    "videojs-contrib-dash": "^2.11.0",
    "viewport-mercator-project": "^6.2.2",
    "xml2js": "^0.4.19"
13

Published an article on the topic:

@waldgeist I will take a look at the packages later on

7 Likes
14

@waldgeist so I checked a bit and manual resolution of the packages is very time-demanding. I was thinking if there is the option to add some kind of debug-logging to the meteor-node-stubs package so we can narrow down first, which node modules were stubbed and then analyze our node_modules folder for dependants on these packages :thinking:

3 Likes
15

Thanks for taking the effort. Yes, that sounds like a valid approach.

16

I find it still extremely hard to debug, I’ve got rid of all my explicit dependencies to crypto but now I have no idea who is importing it :confused: package-lock.json is not saying much, the dependency leads back to meteor-node-stubs. This is probably due to one of the Meteor package being responsible of including crypto?

17

I would like to recover this discussion since I already saw a few times the following question: why do we need meteor-node-stubs

Another motivation to recover this discussion: This pkg ataches the node_modules, what cause a i’m seeing a high frequency of security issues reported due outdated sub-pkgs:

we already is in patch x.x.23, we cant wait to arrive at x.x.203 lol
Jokes aside, I see it’s unnecessarily consuming the maintainers

IMO, we need understand and document with clarity the necessity and usage of meteor-node-stubs , and:
if (it’s really necessary) → we should refactor it to detach the bundled node_modules from the pkg
else → we should remove it from Meteor 3.x.

@jkuester in your tests, do you remember if you just removed it and ran or did you do another extra validation? i’m thinking to perform similar tests removing that pkg for Meteor3.x

@radekmie, @nachocodoner , @grubba do you know something about this module that can help in this thread?

7 Likes
18

From my understanding the node stubs were initially added to achieve full isomorphism with node. However, I actually never really use them because most of the tools are either not used (like FS in browser) or are already natively available (such as crypto with web crypto).

From my end I would handle this package the following:

  • remove it as necessary dependency
  • provide/maintain only the wrapper layer
  • if a user needs a node package being stubbed in client then let them install and maintain it on their project level but let them use meteor node stubs to easy stub without much hassle
3 Likes
19

That would be amazing. It’s a large part of my bundle, and for mobile apps it is essential to keep it small.

1 Like
20

maybe @zodern could help here, Italo…

This is what I would say too!! I think time has passed enough that we have almost a 1:1 browser:node API

5 Likes