Any guide on application architecture for enterprise-scalable applications?

[waldgeist]

I am expecting my application to become quite big eventually, so I am trying to set-up an architecture that can keep up with its growth from the very beginning. Yet, I am struggling to get this done in a really satisfactory way.

I am already using packages to modularize every function group into a separate modules. This kind of works, but of course there are some interdepencencies between these groups, and I find it quite hard to manage these without turning the whole thing into some kind of spaghetti code.

Here are just two samples to explain what I mean:

• I am using SCSS (fourseven:scss) and Bootstrap throughout the application. So I have defined a central Bootstrap theme that sets SCSS variables, e.g. for the brand colors of my application. Of course, the functional groups (packages) of my application depend on these central variables. But I did not manage to set-up this without hard-include the package SCSS files into my central application (details here: https://github.com/fourseven/meteor-scss/issues/104). This may be an issue with fourseven:scss (and may be solvable with the new build process that will be available with Meteor 1.2), but that’s just a sample for a situation where some kind of bridging functionality in the Meteor core would help a lot to keep things clean.

• Regarding namespacing: Nearly all public packages export their own objects to the global namespace. I think this is OK for larger packages or core functionality (like FlowRouter), but for my much smaller application-specific modules I don’t really want to pollute the global namespace that much. Hence, I am using a global “AppMain” object which is defined in one core package all other packages depend on, and then add my package-specific exports to this variable. This works, but in the end this also causes interdependencies I would like to avoid. It would help a lot if it was possible to map package exports to the application-specific namespace instead.

I do not expect anybody to post solutions to these specific problems here, as they should only serve as some examples. Instead, I would be happy if anybody could point me to guides / tutorials / blogs by people who already set-up really complex Meteor applications (i.e. beyond the simple projects included in the Meteor tutorials). The only tutorial I found so far on this topic recommended not to use packages that much; I don’t think this advice will bring me far.

(BTW: I already had a closer look at Telescope. This was the only larger application I found that was available as OpenSource. The structure of Telescope is quite nice and clear, but it is rahter tailored to one specific app. I am looking for an architecture that allows my modules to be easily exchanged across applications.)

[shock]

this is my favourite so far (we are slowly migrating to it atm) - let you scale from 1 all-in instance to microservices with same source
http://meteorpatterns.com/Sewdn/project-builds

[serkandurusoy]

The first point will hopefully have been resolved with 1.2

About namespacing, you might want to look into other modularization approaches. There’s some good discussion on these threads in case you have missed them

• New package "rocket:module" to provide CJS/AMD/ES6 Modules for Meteor packages with shared NPM dependencies
• Universe Modules: use ES6 / ES2015 modules in Meteor today!
• Meteor + Webpack: ES6 modules, hot-code-patching, fixes load order & more

[vjau]

Regarding namespacing: Nearly all public packages export their own objects to the global namespace. I think this is OK for larger packages or core functionality (like FlowRouter), but for my much smaller application-specific modules I don’t really want to pollute the global namespace that much. Hence, I am using a global “AppMain” object which is defined in one core package all other packages depend on, and then add my package-specific exports to this variable. This works, but in the end this also causes interdependencies I would like to avoid. It would help a lot if it was possible to map package exports to the application-specific namespace instead.

Correct me if i’m wrong, but when you are importing a public package A in one of your package B, all of package A exports are local to package B and don’t pollute the global space.
AFAIK, there is no way for a package imported by another package to pollute the global scope (?)

EDIT: they could probably put their vars on “global” or “window”, but that’s not the usual way to do export in Meteor, is it ?

[waldgeist]

@shock: Thanks for linking this article. This is exactly the kind of best-practices articles I am looking for. I like the idea of having “umbrella” packages, this reduces repeated meteor adds across projects. Also appealing to me is the idea of splitting the app into separate, scalable services. Unfortunately, the article does not really explain how these different parts communicate with each other (is it really based on just using the same database, as the npm config script implies, or is there more involved like DDP?). It would be interesting to read a bit more about this kind of scalability.

[waldgeist]

Hi @vjau, I did not mean that the public packages are polluting my namespace, in fact they don’t, as you described. I meant that I do not want my own packages to pollute the global namespace too much, since they are smaller than public packages (they only contain functional modules), and I don’t want to risk name collisions with public packages that I am not aware of now but might be using in the future. This is the main reason why I am at the moment encapsulating all my exports in one global AppMain object. But this just does not “feel” very nice, since it also incurs dependencies between the packages.

[waldgeist]

@serkandurusoy: Thanks for picking these links for me. Will have a closer look at them now!

[dalgard]

Dependency management is not an easy subject. It will take some trial and error to get the right dependency tree for your application.

Don’t be afraid to try something out and then refactor, maybe more than a couple of times. Congratulate yourself on having things divided in packages in the first place. It will help you stay on the straight and narrow path, even if it sometimes feels like a Gordian knot.

Try not to rely on imply and the global namespace – only use the things that you’ve imported with use.

[SkinnyGeek1010]

I am expecting my application to become quite big eventually, so I am trying to set-up an architecture that can keep up with its growth from the very beginning. Yet, I am struggling to get this done in a really satisfactory way.

I haven’t build a truly large app yet but the largest is around 50k LoC and several hundred files. Here’s how I do it to solve my own pains of maintainability:

modules

It’s unfortunate that Meteor currently offers no way to build large apps, as everything has a global namespace. At best each package exposing 1 global each. There is also no wiring to de-couple business logic from views.

To fix global namespaces and dependency order i’m using Webpack to use ES6 modules. This keeps everything nice and modular with no globals leaking everywhere (you can’t even do a Posts.insert in the console if you wanted to). If you build them correctly they should be able to be used in any app and have little dependancies.

I also use React to help make UI modules that are mostly pure functions with the exception that some depend on a single dispatch function that the view calls when something happens. They also accept a single state object as a parameter that has the entire app state (which is immutable by convention). This makes them very easy to test and even easier to drop into other apps (I share some across 2 apps).

Meteor packages still work as they’re global… your code just assumes they’re there as usual.

By using Webpack and NPM you can split out your common code into private NPM modules that only your team can use (you could also just use relative paths and require them in package.json).

Decoupling

A large app can really benefit from decoupling any logic from the view. Redux is a popular library to help structure your code so the view just dispatches an event with some meta data. It’s based off the flux architecture and the Elm language. It works with any view framework and I recently made a spike to work with Blaze using Reactive-Dicts. (though I use it with React on a day-day basis).

I’ve used a ‘services’ folder that has functions for business logic that both the client and app can consume. If there’s a group of functions in the same domain then i’ll call an object bag something like UserServices and then call UserServices.createUser These are pure business logic and the ‘action creators’ from Redux typically fire these off to produce side effects like sending a welcome email or creating an account.

These services should also be very easy to port over to a separate meteor app as a service if needed. I have 2 that I split off into separate headless meteor apps that just handle a scheduling queue and another just handles notifications (mostly for performance reasons with Node’s single thread).

My app structure looks like this which makes code sharing easy:

  ---| App
    ------|  Actions
    ------|  Components (views)
    ------|  Pages (components that are 'pages' for router)
    ------|  Routers (client and server routers)
    ------|  Services
    ------|  Styles
    ------|  Utils
    main.client.js   (entry point of the app)
    main.server.js   (entry point of the app on server)

If this interests you checkout this for the info on webpack:
Meteor + Webpack: ES6 modules, hot-code-patching, fixes load order & more

and here for the info on Redux/flux architecture and an example (use redux branch)
Redux for Blaze Example App
Is flux worth using. When is it good or bad to use
https://github.com/AdamBrodzinski/meteor-flux-leaderboard/tree/redux

and here for an example of decoupling the UI similar to Redux without using it: https://github.com/adambrodzinski/react-ive-meteor

[waldgeist]

@SkinnyGeek1010: Thanks a lot for this detailed info. It seems as if I should really delve into React and Webpack. Well, I am learning so many frameworks these days that I do not really succeed in driving my app’s functionality forward. That was the only reason why I was still reluctant to jump into React.

[SkinnyGeek1010]

Well, I am learning so many frameworks these days that I do not really succeed in driving my app’s functionality forward. That was the only reason why I was still reluctant to jump into React.

I totally understand where you’re coming from. It never seems to end

If you do decide to go with React it should only take a weekend to feel really comfortable. Since time is a factor I would recommend this video series… you’ll be up and running in no time (as well as using a basic webpack config). They also have some others React series that are a bit older. There’s also a flux tutorial that explains the architecture (note Redux is much simpler but the video at least is worth a watch since Redux is based on flux).

Build Your First React.js App

I also made a quick webpack for meteor screencast in the first link, it’s not comprehensive but good enough to understand what’s going on.

[waldgeist]

@SkinnyGeek1010: Thanks for the link, seems as if I need an egghead account now as well

BTW: Another reason why I was reluctant to go for React is that I love the clear separation of HTML and JavaScript that technologies like Blaze provide. At the moment, I am a single developer. But if the team grows, it is quite sure that there will be people involved who are not developers themselves and “only” know HTML and CSS. To me, React seems a bit like technologies like Google’s GWT in that it moves front-end development completely to the “backend” developer. What are your experiences regarding this separation of concerns?

BTW: I like the idea behind Blonk, simple and convincing

[SkinnyGeek1010]

Thanks for the link, seems as if I need an egghead account now as well

Yea they’re really good! However they’re organized terribly. I often search by author or search on Google. They have tracks like Reflux (not Redux… yet), and others that are not listed in their “tracks” section.

What are your experiences regarding this separation of concerns?

The only issue I have with it is that it takes a bit of time to onboard new designers. It’s more of an issue if you outsource someone to do your HTML/CSS as they prob. won’t work with it.

However the easiest solution is to give the designer a zip file with sass and a shell with the footer/header and let them code the inner html/css and then you can copy/paste the html and run a find/replace on class.

I’ve always had the designers give me sketch/psd files and then I had to do the markup myself so that wasn’t a problem. Also if I had an in-house designer that would do html/css then I would just take the time to show them how to use jsx… as it’s nearly identical to html.

If you think about it all they have to do is write:

Page = React.createClass({
  render() {
    return (
      <div className='Page'>
        Hello
      </div>
    );
  }
});

// instead of this:

<template name='Page'>
  <div className='Page'>
    Hello
  </div>
<template>

And also in the next release you can even write a stateless component like this:

Page() {
  return (
    <div className='Page'>
      Hello
    </div>
  );
}

Which is not much more of a jump than the blaze templates. Maintaining a component shouldn’t be bad if things are named nicely (they’re better at it than engineers too!).

Anyhow on more of a personal note with separation of concerns here’s my take on it:

The current state of HTML templates is that they are separated by a file but they still share a concern of the template logic (JS). They have lots of helpers which are often vague when looking at just the template (what does each this refer to again???).

Templates need to be dynamic so we try to add in logic by using {{each}} and {{if}} as well as helper functions. These end up destroying the pure HTML. You also need to jump into the JS file to edit these and often I have a vertical split with both JS and HTML open to track what’s going on (alwyas when debugging).

These two things are very coupled with each other most of the time. It’s rare that I can take a blaze template and use it on another part of the same app without copying and pasting the JS out of the old JS file. It’s also hard to make these similar but not duplicated (e.g. they don’t compose easily).

[quote="waldgeist, post:12, topic:9431"] To me, React seems a bit like technologies like Google's GWT in that it moves front-end development completely to the "backend" developer. [/quote]

To me it seems more like the front-end role for the web is changing. In the past we had thin clients. Front-enders (was me 4yrs ago) only had to worry about HTML, CSS and enough JS to make tab containers work. However now web-apps are ‘thick clients’ just like Android and iOS app. Most of the logic is on the client and the backend is mostly a JSON pump.

Handlebars worked great 4 years ago when it was just in the PHP or Ruby ‘public’ folder. However if you want to do complicated things like a thick clientside app you start to really want more control over how that view is handled.

React allows me to think about separating components in my app. Nested pieces of UI that can be thought about on their own (adding simplicity). By putting all the HTML/JS in one class/function, I can easily move it around and compose higher order things like creating a Label and Radio input and then using those to compose a new RadioYesNo component that just takes a label value property:
<RadioYesNo label="Do you like Bacon?" />.

React also brings another cool thing to the table. An entire ecosystem that embraces simplicity over complexity, re-usability, and composibilty.

When I go back to my legacy apps with Blaze I now think… ughh this is so messy.
Just my $0.02!

[waldgeist]

Thanks for these insights!

[jcarson]

May be it is better to write components of React instead of many Meteor packages. At present the focus of using React is on UI design. I prefer to use Blaze/HTML5/CSS for UI but React for components. Unlike Angular React can be smoothly integrated with Blaze.

[waldgeist]

That’s an interesting approach. Is it possible to use React like this in Meteor 1.1, or do I have to switch to the yet unstable 1.2 then?

[jcarson]

It worked for 1.1. need to add react-template-helper package.

[SkinnyGeek1010]

That’s an interesting approach. Is it possible to use React like this in Meteor 1.1, or do I have to switch to the yet unstable 1.2 then?

Yep you can do it now. I made a repo React-ive-Meteor that uses this approach (hybrid of blaze and react). The outermost shell is blaze and the insides are React.

However in hindsight it’s more work (if you’re starting from scratch) and i’ve switched to 100% react. Still a very nice way to migrate if you have Blaze in prod. In fact this is a great video on refactoring a current code base: https://www.youtube.com/watch?v=BF58ZJ1ZQxY

[sircharleswatson]

The author of that post that @shock referenced is working on a follow up article to talk about how to connect all of those different microservices. I asked him about it on twitter. haha