Building multiple apps from the same source

We (http://redandivory.com) have gone though different approaches in the past few years to build multiple meteor apps from the same source base. Most of the past approaches used concepts like git submodules or private smart (meteorite) packages to share private app-specific packages over different meteor projects.

We are currently looking at a new approach, where we don’t need to separate common library-code to a different repository and don’t need to setup a meteor-project for a new build.

First, let me elaborate on the use cases for having multiple meteor builds (and i don’t mean builds for different platforms, but builds for different uses in your platform stack).

Use Case 1: I want different user groups to use different apps.
Say you have an app with completely different enduser experience depending on your role. It is common practice to have the user loggedin, and setup different routes based on the user’s role, and load different templates to serve that type of user’s needs. But all types of users, load the same app, and the app decides which portions of the build to use and not to use. This results in builds that can be way too large for a user with a role that can actually do very few things with your app. A common example: a unauthenticated user just needs to be shown the landing page, must be able to login and to register a new account.
Furthermore, this approach leads to security vulnerabilities. Because all of the logic is deployed and catered to all users, whatever their role, you must be damn sure your access rights are checked all the way through in every aspect of your app. It would make more sense, just not to load the code for users who don’t need it or shouldn’t use it.
To support these use cases, one should deploy different apps to gewgl.io, app.gewgl.io, admin.gewgl.io, … and make sure the right user uses the right app.

Use Case 2: I want my backend to be made up of microservices
Say you have a very computation heavy process somewhere in your backend and you know this is going to have scaling issues as soon as your userbase grows. You want to be able to separate this process from the rest of your app, deploy it as a independent service and scale it seperately as your userbase grows.
Of course it is possible to set this process up as a different service, and make your meteor app communicate with it as with any other 3rd party service (or even better with DDP, since their are both meteor). But if you want to do this for 15 different components within your backend, this process of building different services can get very tedious.
It would make more sense to build all the services in your single app during development, but build separate apps for all the different microservices as soon as you build for production.

To support both of these use cases within the core of Meteor, you should be able to control the contents of .meteor/packages when building your app for a certain context. The problem now is your packages file is fixed for the current state of your app and you can’t change this packages file during build, because another meteor app might be running with it and reloads as soon as the file contents is changed.

So I’m basically looking for the support of multiple .meteor/packages files to be able to build different apps with other package dependencies from the same meteor project.

Does anyone know of an approach that is currently supported with meteor core, that results in the same benefits?
Does anyone know if MDG has support for these use cases on their roadmap?

Thanks for reading this far :wink:

Kr,
Pieter Soudan

4 Likes

@sewdn - We have very simmilar use-cases. As of now our app is in development, and we have been succesfuly using the sym-link technique - meteor run is working fine. We have issues with meteor deploy, we are researching on it.

Yes, i use symlinks too. I had an “Aha-Erlebnis” when i had written this post. Verbalizing the problem is the start of the solution :wink:

The way I’m am doing it now:

  • I have 1 big repo for my entire platform
  • every app i want to build (user-apps, microservices) is a separate meteor project with its own directory containing the app-specific .meteor directory
  • I setup a packages directory at root level of my project, and I symlink to this directory from every single app-directory
  • I organize my platform so every app is made up of app-specific packages that are shared using the symlinks
  • every app (that needs individual deployment) has its own umbrella package implying all of its root dependencies.
  • only the umbrella packages are used in the different .meteor/packages files

This works very well for now.

3 Likes

You should make packages, some interconnecting code that registers packages and use settings.json to manage this.

Reaction Commerce has a fantastic way of managing this here: https://github.com/reactioncommerce/reaction-core/blob/05a645877b876a1cceb9ca6fe6d293db2ee89a78/server/register.coffee

Please avoid using symlinks I think you can add more than one directory to the env variable PACKAGE_DIRS to achieve the same effect.

Symlinks are bad.

2 Likes

Could you please tell us why symlinks are bad in your opinion?
I don’t see any harm in using symlinks to share a set of packages over different apps within your project.

I wrote an elaborate post on the subject of building multiple apps for a single project from the same source base: http://bit.ly/1NRjDla.

Please comment on this post in this thread!
Thanks!

1 Like

Symlinks require extra overhead and scruitiny when moving to a scaled environment.

Are all your symlinks relevant and valid?
Do they point to the correct place?
Do all tools and software treat them the same?

What if one needs to be updated?
What if they turn recursive? Recursive symlinks are a nightmare.

They also lead to confusing setups for development teams.

But mostly because. There’s an environment variable in place for you that can do the job of these symlinks and you don’t need to manage them.

I’ll read your post which has more details and get back to you!

/project
  /app
    /.meteor
    /packages => ../packages
  /landing
    /.meteor
    /packages => ../packages
  /services
    /.meteor
    /packages => ../packages
  /packages
    /myproject:core
    /myproject:app
    /myproject:app-profile
    /myproject:app-projects
    /myproject:app-ui
    /myprojects:services
    /myprojects:landing

Yeah use PACKAGE_DIRS to point to that top level package directory as an environment variable. Much cleaner and predictable. Supports seperated builds too!

2 Likes

So, is there a fresh link to the github article above? I tried it and it doesn’t resolve, but I too would like the best way to build multiple different user role apps separately within my single meteor project.

You can look at this article: http://meteorpatterns.com/Sewdn/project-builds

But for Meteor 1.3 you’ll need to revise this pattern by declaring your module dependencies differently.

1 Like

Thanks, I’ll check it out.

And, yes, I’m a little worried, as I’m learning Meteor 1.2.x, and 1.3 looks like it’s going to change a lot of the ease that I’m enjoying in 1.2.x…

You can stay with Meteor 1.2.x style of app if you want as Meteor 1.3 is backwards compatible. However, taking advantage of the new ES6 modules support is key if you want to have the best experience with building multiple apps from the same source.

We were going down the package approach and refactored to using modules with import/export and it is great. We are building web client, cordova mobile app, common server and microservice server all from the same code base.

2 Likes

It sounds awesome! I’m just not a professional developer, so it takes me a while to get the hang of something. My normal pattern is…become really good, realize I’m 10 years behind. Pick up something new…repeat.

@skirunman and others, can you say more about your project structure? I’m looking for the clean 1.3 way of doing the multiple app thing. Any help would be much appreciated.

4 Likes

Yes, I’m also looking for the clean Meteor 1.4 way of having multiple apps from the same code base. It sounds like its @sewdn’s article but using modules instead of packages? Is there a simple example of this anywhere? I’m refactoring my Meteor 1.1 app at the moment.

This is the structure we use. We run three Meteor apps on Galaxy, a main app and 2 “micro-service” apps that are server side only that are called from the main app. All 3 apps are checked into the same git repository. There are a few key things that we do that makes it easier to reason about our app and also make it easier to refactor code. We have a very extensive design document that covers many more details such as naming conventions, etc. but I don’t want to post it here:

  1. We always use ES6 imports and exports to control code load order including with global code such as Meteor packages, import {Meteor} from 'meteor/meteor';
  2. We use an index.js, which imports all files and re-exports the public APIs, for all our modules and certain directories where we want to be able to easily import the entire functionality of that module or directory.
  3. We use relative imports within our modules or functional areas like import foo from ‘../foo’; and absolute imports across modules like import foo from ‘/imports/api/foo’;.
  4. All the re-usable non-UI code is in /imports/api/ in the main app directory. The other apps have a symlink from their own /imports/api/ directory to the main app.
Meteor App high level directory layout
app/			# a Meteor app directory, multiple app directories are supported
builds/			# app client and server builds, if any
config/		        # settings.json and env variables for dev, staging, production
scripts/		# misc. scripts for build or testing, if any

A Meteor app directory layout
.meteor/                # Meteor package configuration and local builds
client/
  main.js               # client entry point, imports all client code
  main.css/             # imports other styles from imports/client/components, if needed
  main.html	        # tags in <head> or use package to inject like react-helmet

server/
  main.js               # server entry point, imports all server code
 
imports/
  startup/		# All app-wide, common, and startup configuration
    client/             # configuration for routes, subscriptions, or any other client side services

    server/		# configuration for users, email, oauth or any other server side services
      index.js		# imports all server bootstrap, security, config and api code
      bootstrap.js      # general configuration startup code
      security.js       # set browser and security policies
      register-api.js   # imports server only code from api/*/server and all other api code
      user-config.js	# configure user account profile and email templates

  api/			# business logic
    moduleOne/          # a unit of business logic
      index.js		# imports all module one apis and re-export public apis for outside use
      ModuleOne.js      # module one collections
      methods.js        # module one methods
      schema.js         # schema definitions for module one
      ModuleOne.tests.js # tests for module one
      server/           # module one code that runs only on the server
        index.js        # imports all server code for module one
        methods.js      # server only methods for module one, if any
        publications.js	# publications for module one

    lib/		# api utility functions and shared libraries

  client/               # client only code so changes won’t rebuild server code
    components/	        # reusable ui components
      ComponentOne.js   # contains simple, stateless React component
      componentOne.tests.js # tests for component one
      ComponentTwo/     # complex React component with subcomponents
        ComponentTwo.js # main React component which imports subcomponents
        ComponentTwoContainer.js # HOC React code to fetch data and pass state to component 
        ComponentTwoA.js # stateless React ui subcomponent
        ComponentTwoB.js # stateless React ui subcomponents
        index.js        #  imports all component two ui code and re-exports

    layouts/            # wrapper components for behaviour and visuals
    lib/                # ui utility functions and shared libraries
      index.js          # imports all lib functions and re-exports as default and for each function
    styles/		# global style definitions including 3rd party vendor styles

  packages/		# 3rd party packages that we need to modify and support, if any

node_modules/           # app NPM packages defined in package.json
public/			# fonts, images, icons, favicon
resources/		# mobile icons and splash screens
tests/			# integration tests not run by Meteor built-in test tools

EDIT: I should say that we support both web browser and Cordova mobile apps on iOS and Android from the same code base. For the UI we are using Material-UI. We have created some very basic layout components to deal with the size/layout differences for browser and mobile, but this is very little code to achieve what we need. We also had to use a different list component to get better performance on mobile, react-virtualized.

3 Likes

Looks like your article is a gold mine to solving the problem. Any new links to the article you posted before? Thanks!

1 Like

Looks like the article can be found here