I’m trying to plan the architecture of a Meteor app for speed and scalability using a microservices/event-based model. I will also be hosting the apps in Docker containers. I have questions concerning implementation and other “gotchas” I should consider.
BACKGROUND
Several commentators have suggested a split application model for creating Meteor apps: Dan Dascalescu (www.youtube.com/watch?v=aYIK_ClHLZo), Arunoda Susiripala (bulletproofmeteor.com/architecture/microservices-with-meteor-and-ddp), Red and Ivory (meteor.redandivory.com/#). This would reduce the size of each “microservices app” that is called and presumably would lead to faster load times.
The overall app would be split into multiple microservices based on features such as landing pages, login, profile, commerce, newsletter and so on. Since the landing/marketing pages (Home, About Us, etc.) do not need to be dynamic, I plan to create a separate app to render these pages as static pages as suggested by Percolate (blog.percolatestudio.com/engineering/architecting-better-landing-pages/).
From a hosting perspective, I would use the Cluster package (https://github.com/meteorhacks/cluster) to allow some Meteor microservices apps to be scaled more so than others. All the apps would be housed in individual Docker containers and orchestrated using a Docker orchestration tool such as StackEngine or Tutum.
IMPLEMENTATION ISSUES
1. Separate Subdomains and Sharing Login State
Presumably, every app would need to be in a separate subdomain (such as app1.example.com, app2.example.com, app3.example.com). There is a new Login-State package allowing for the sharing of login state between subdomains. (See kadira.io/blog/sharing-meteor-login-state-between-sub-domains/.)
Would the Login-State package work for apps in a cluster using the Cluster Package, or would cluster-wide authentication still be required? (See discussion at github.com/meteorhacks/cluster/issues/14.)
2. DDP for Event-Based Message Queues
Would DDP be sufficient to handle events/messages among the Meteor microservices apps or would some external queue service (such as RabbitMQ or AWS SQS) still be helpful/necessary?
3. Cluster Package (alone or with other services)
a. Service Discovery. Would the Cluster package be sufficient to handle discovery of the Meteor microservices apps in Docker containers, or would an additional service discovery tool such as Weave be helpful/necessary? (See weave.works.)
b. Load Balancing/Failover. Would the Cluster package be sufficient to handle load balancing/failover, or would an additional load balancer (such as Nginx) be helpful/necessary?
c. Multiple Hosts. Would the Cluster package work across multiple hosting providers?
d. Scaling Specific Clusters. Can the Cluster package scale specific groups of apps only (such as the “ecommerce cluster” or the “newsletter cluster”) if there is high traffic to just these clusters?
e. Docker Container Scaling. Can the Cluster package be used to manage the Docker container scaling as well or would this need to be a two-step process as in: (1) set up Docker containers first; and then (2) scale Meteor apps in those new containers using Cluster? Alternatively, would scaling the Docker containers using a separate tool also scale the Meteor App clusters?
4. Load Balancing Using Docker Across Web Hosts
Assume the apps are hosted in two different web hosts: one on the East Coast of the US (Web Host A) and the other on the West Coast (Web Host B).
a. Scale Per Host. From a performance perspective, should I create a rule that a microservices app cluster in Web Host A should only link to a database cluster in Web Host A (with the database cluster in Web Host A synching to the database cluster in Web Host B in the background)?
b. Centralized DB Cluster. Would one centralized location for a database cluster be preferable instead? In other words, should a database cluster be hosted in Web Host C, with Meteor microservices apps in Web Host A and B pointing to that Web Host C (database cluster)?
5. Session Management
Beyond using either tokens or other persistent session package, are there any other session management issues to consider when creating clusters across different hosts?
Can you think of any other issues or “gotchas” to consider in implementing the above approach?