Opening new thread rather than add caveat defending smoke signals ( 
) to the @kfritsch post @leonardoventurini blew wide open: Reactivity problems after migrating to meteor 3.0.3 - #7 by leonardoventurini
After implementing BullMQ ( which went great ) per: What do you use for job management in your production app?
Instead I would ask: Are there established ways to add new first-order entry-points that run on the same host as server or otherwise run in containers if not on the same host?
client is running on an external machine and linking up to the server instance, by convention, but what about another entry point being added, expected to run under administrator control?
And not just one necessarily, but perhaps an entire cluster. Trying to put it in tangible terms, in pseudo for now, starting from nuts and bolts, and not factoring in Galaxy implications at all. Also I assume Docker because of a long explanation I will omit.
In package.json I would see something like this, pardon any errors in the format of that file:
"meteor": {
"mainModule": {
"client": "client/entry.js",
"server": "server/entry.ts",
"agent/type1": "agent/entryType1.ts",
"agent/type2": "agent/entryType2.ts"
}
From that example, beyond client which is not-local, we have three entry points which are complete applications which might or might not run, but at least be built. server would run like now, but other entry points might only be built and be available.
This would be using the imports/ behaviors where the individual builds only include what is explicitly imported, for the smallest possible footprint.
Lastly, as with meteor-base now ( not using mup myself but curious about that ) … there might then be specialized Dockerfile.agent1 and Dockerfile.agent2 which might then yield different images that can be distributed across machines or run on the same virtualization host.
If all that happens, by way of BullMQ for example, now there are not only “runners” or “workers” but full-fledged agents, which fulfill a certain function or range of functions. That might include background processing of “jobs” but it is more likely to be not as basic. Such as a Matrix connection or a protocol listener. Then there could be a tiny ( do not think “microservice” ) agent with a very specific focus, and kept alive by the containerization platform, with Docker in my case.
Agent is a versatile term because some will interpret this to be an bot running, whereas others might program specific behaviors with permissions of a particular person with credentials. It could be code supporting wo/man or machine but still be “Agent” and the term would hold.
Anyway, the only root requirements are:
- Declare more entry points that are built for infrastructure rather than be run externally
- Those get optimized and built exactly like
serverwould - Those builds can be imaged individually to containerize only the entry point desired
A lot of this is really an “omnirepo” type code insanity minimization technique because nothing stops one from individually building separate repositories / packages, but as long as there is optimization of the build, if/since all “entry points” ought to be in the same code context, there can be a shared package.json as above.
Right now I have an embedded / red-headed-stepchild package for agent/entry.ts and it is only part of the repository, but not part of the same cohesive system plan, at the builder level.
So to park my bike in the shed better on the inspiring post: “smoke signals” being “the first telecommunications” could have been pit against “bonfires at night” or bells within a certain range under certain conditions, or even carrier pigeons, and the point mostly holds.
But those are still lower latency than horses and sneaker-net ( walking on feet with shoes on across distances ) … but “smoke signals” are more than a 1 or 0 and do not require physical movement of a message but long-distance reception instead, and are not packaged, and have high resolution information. And all these “use the natural world” rather than “serve the purpose of life” directly on bare metal, like lightning does, where there is a balancing out taking place by that “message traffic” … We have tons of these. Individuals and communities have SO MANY “obvious messages” which are more than 0 or 1 and fit more into the style of communication we do now, every day, with message queues.
My hope here is to make the same type of shift that happened in achieving airplane travel, where rather than flapping our wings at scale, we discern the underlying physics and rethink the entire problem. From there, I think Meteor is at the cross-roads of many massive trends over very long swaths of time, and it would be great if we could make a leap to be future-back, rather than past-forward in terms of design. Right now it feels like we chase the Jones in design, or do not do design at all really, when we can very simply see what we are, what we are doing, and design around that case. Would be awesome to brush away all the layers of ladder up to here.
… and I even pasted the code above 
Just wow. Oh well, much more acclimated 
What a monster.