Special purpose entry points beyond client & server?

help
1

Opening new thread rather than add caveat defending smoke signals ( :rofl: ) to the @kfritsch post @leonardoventurini blew wide open: Reactivity problems after migrating to meteor 3.0.3 - #7 by leonardoventurini

After implementing BullMQ ( which went great ) per: What do you use for job management in your production app?

Instead I would ask: Are there established ways to add new first-order entry-points that run on the same host as server or otherwise run in containers if not on the same host?

client is running on an external machine and linking up to the server instance, by convention, but what about another entry point being added, expected to run under administrator control?

And not just one necessarily, but perhaps an entire cluster. Trying to put it in tangible terms, in pseudo for now, starting from nuts and bolts, and not factoring in Galaxy implications at all. Also I assume Docker because of a long explanation I will omit.

In package.json I would see something like this, pardon any errors in the format of that file:

  "meteor": {
    "mainModule": {
      "client": "client/entry.js",
      "server": "server/entry.ts",
      "agent/type1": "agent/entryType1.ts",
      "agent/type2": "agent/entryType2.ts"
    }

From that example, beyond client which is not-local, we have three entry points which are complete applications which might or might not run, but at least be built. server would run like now, but other entry points might only be built and be available.

This would be using the imports/ behaviors where the individual builds only include what is explicitly imported, for the smallest possible footprint.

Lastly, as with meteor-base now ( not using mup myself but curious about that ) … there might then be specialized Dockerfile.agent1 and Dockerfile.agent2 which might then yield different images that can be distributed across machines or run on the same virtualization host.

If all that happens, by way of BullMQ for example, now there are not only “runners” or “workers” but full-fledged agents, which fulfill a certain function or range of functions. That might include background processing of “jobs” but it is more likely to be not as basic. Such as a Matrix connection or a protocol listener. Then there could be a tiny ( do not think “microservice” ) agent with a very specific focus, and kept alive by the containerization platform, with Docker in my case.

Agent is a versatile term because some will interpret this to be an bot running, whereas others might program specific behaviors with permissions of a particular person with credentials. It could be code supporting wo/man or machine but still be “Agent” and the term would hold.

Anyway, the only root requirements are:

  1. Declare more entry points that are built for infrastructure rather than be run externally
  2. Those get optimized and built exactly like server would
  3. Those builds can be imaged individually to containerize only the entry point desired

A lot of this is really an “omnirepo” type code insanity minimization technique because nothing stops one from individually building separate repositories / packages, but as long as there is optimization of the build, if/since all “entry points” ought to be in the same code context, there can be a shared package.json as above.

Right now I have an embedded / red-headed-stepchild package for agent/entry.ts and it is only part of the repository, but not part of the same cohesive system plan, at the builder level.

So to park my bike in the shed better on the inspiring post: “smoke signals” being “the first telecommunications” could have been pit against “bonfires at night” or bells within a certain range under certain conditions, or even carrier pigeons, and the point mostly holds.

But those are still lower latency than horses and sneaker-net ( walking on feet with shoes on across distances ) … but “smoke signals” are more than a 1 or 0 and do not require physical movement of a message but long-distance reception instead, and are not packaged, and have high resolution information. And all these “use the natural world” rather than “serve the purpose of life” directly on bare metal, like lightning does, where there is a balancing out taking place by that “message traffic” … We have tons of these. Individuals and communities have SO MANY “obvious messages” which are more than 0 or 1 and fit more into the style of communication we do now, every day, with message queues.

My hope here is to make the same type of shift that happened in achieving airplane travel, where rather than flapping our wings at scale, we discern the underlying physics and rethink the entire problem. From there, I think Meteor is at the cross-roads of many massive trends over very long swaths of time, and it would be great if we could make a leap to be future-back, rather than past-forward in terms of design. Right now it feels like we chase the Jones in design, or do not do design at all really, when we can very simply see what we are, what we are doing, and design around that case. Would be awesome to brush away all the layers of ladder up to here.

1 Like
2

I like the concept of “agents”.

I don’t think there is a proper way to do what you are saying, right now, in a single Meteor project.

It would be interesting to support agent entry points; perhaps that could even help support standard/built-in agents that could offload reactivity tasks, for example.

I certainly agree that we need a paradigm shift in Meteor, a breath of fresh air, and it’s been a long time coming. Meteor 3 is here and now we can make it even better than before.

Will give this some more thought and discuss with the team. Are there any resources you can recommend on this?

Would love to see what you build!

Thanks!

2 Likes
3

Pursuing the paradigm shift, and pulling in comments to @rjdavid on the first linked post, I see this about to happen on my end. Also I am commenting toward psychological state here, since most of the hurdle here is in our heads. Not resolving complexity, but overcoming our own best.

  1. Fork or abstract Meteor so I can have multiple agents, hoping to send code back, or perhaps not use a fork if it gets absorbed ever.
  2. Continue to wrap and not use Meteor ( or even BullMQ ) first-class objects as much as possible. Sugar is a food group in design.
  3. Take an opinionated course forward for agility and momentum, and hope there is enough diversity and tolerance here to not cause flamewars and bikeshedding, hoping to stress out the overlapping point, which is the flexibility of Meteor itself, since Skiers and Snowboarders both carve the same mountain
  4. Remove all extra build processes, all extra repositories, all redundant packages, etc… DRY everything at the macro level, because right now for example, I would rather kill myself than maintain the number of repositories it would take to cover the requirements of the design. Once you have one agent, you find yourself having many many agents, and start with big heavy agents and then spin off parts into more focused agents… but if that is more than just making a new entry point, the cost of developer sanity is ridiculous and “cruel and unusual”
  5. Deploy flagship implementations of “light up all cores” type systems with all the functionality shown, not discussed, but demonstrated… I feel like that is the end-game for Meteor … I would close on that most important part:

I am not all the way through that thread yet, but I am a solo-founder of >20 currently active/strong companies, who directly guides other founders, and is working in strategy, design, development, even down into management and operations. When I run across Meteor I feel like I found a broken Excalibur in a stone, useful to turn the tide in my radical situation as a principal. That is to accept this is broken but still see it is better than a knife in a gun fight. Independents have stuff to not just get done but never have come up again and we go on killing sprees, called coding. Meteor can fit into the “solo-founder unicorn” demographic much more easily than anything else.

That is the real demographic starting to pop up. Independents with real problems. Overcoming impossible, every shift. Not just startups, which are important, and might become independents. We are taking massive workloads that normally would mean “liquidate or fire everyone” ( or never start ) and slicing through it all because FINALLY there is a non-insane approach. That is superman versus squirrel. That is different from “hot app” or the level of hype people usually capitalize on… versus moving from my arms and back to a crane that takes 5 city blocks but can make a skyscraper. I emphasize the value of this one change for more than effect.

I feel like if we can brush off the “where are we” or “where did we come from” or “where are we going” and just beat the hell out of everything else by being the most awesome because that is just authenticity, then this time next year… we might have time-traveled 5-10 years… at the level of will. We are in a “professional athlete” or “sports franchise” therapist type situation, where the titles are at risk because of PEBKAC. Design is that level of challenge and more, always, never less.

We are at the turning point of a trend started in the 60s-90s, like what is planted in one season yields in another… I see Meteor capable of being in that, rather than a link we leave behind. It feels like a good time to check whether there is fire in the veins here and we raise the bar in the field because we can, or seeing that the next era must come… otherwise no way am I going to deal with the new enormity. Kinda feels like the original problem domain ( I hear ) all grown up, much deeper.

Thanks again for your investment @leonardoventurini

4

On #3 above, my direction is Vue.js on the client side, built with Vite in development, so that has me extremely grateful to @jorgenvatle for meteor-vite and vite-builder

This recently started to peek through in a roundabout way here, flipping a bug/issue into a feature next: Entry points not using Vite.resolve.alias definitions. · Issue #198 · JorgenVatle/meteor-vite · GitHub

Rather than post an issue there, I note that here, and I will be needing to trace through the Meteor pieces all the way down into Vite for this to really work, and also resolving spawning of more than one process during development, rather than need to build containers every time “save” is pressed in the IDE.

5

I do something similar by using babel module resolve


...
"env": {
"USER": {
         "plugins":[[
            "module-resolver",
            {
                "alias":{
                  "@admin-router": "./client/router/adminRouterEmpty",
                  "@server-includes": "./server/bundles/server_includes_user",
                  "@main-router": "./client/router/webRoutes"
               }
            },
            "env"
         ]]
      },
      "ADMIN": {
         "plugins":[[
            "module-resolver",
            { "alias":{
                  "@admin-router": "./client/router/adminRouter",
                  "@server-includes": "./server/bundles/server_includes",
                  "@main-router": "./client/router/webRoutes"
               }
            },
            "env"
         ]]
      },
}

And then setting the BABEL_ENV value in the environment of the server

Its a bit extra annoying because metoer doesn’t support babel.js configs and you have to do funky stuff in a json to get it to work

2 Likes
6

That looks like moving toward two clients @schlaegerz? :rocket:

Since here:

I have been always updating .babelrc and vite.config.ts and tsconfig.json in their respective format, to be able to use universal path aliases.

In that issue linked though, meteor-vite stopped obeying packages.json for where to get the entry points, and even started inserted code ( treating server as if it was client ) which then made me lose that feeling like this cannot ever be changed, since clearly @jorgenvatle is doing his own builds, kind of “over the top of” Meteor and not yet at the point of production.

( I believe? Not sure anyone is using Vite to build production. I feel like there is a dark cave here still. )

7

Another little “gotcha” so far, which rolls into another issue I have duct tape holding together right now:

agents need settings.json ( or equivalent; context configuration ) but not Meteor itself.

However Meteor is what loads Meteor.settings by magic, and cannot be done manually. But obeying that convention would be radical overkill to import just to get a JSON object.

And/then, client and server builds draw upon Meteor.settings from within Meteor would have no problem, while the agents break.

Leads over to probably removing dependence on Meteor.settings and doing a separate configuration JSON, or loading the same settings file, but accessing it without Meteor.settings

8

Looking deeper into this initial nit of unified configuration values, it does not appear Meteor pulls in the JSON itself, from a file, ever, for settings.json but that it is just a convention which loads configuration values:

That is probably wise, since handling files differs so much in weird cases. But this is a highly opaque and contested area to hit right away. When shifting agents to use settings.json it seems like I too will rather pull in JSON, stash it in an environment variable, then have the agent pull and parse JSON from process.env … feels dirty but checks the box on “Make it work.” for now.

meteor-base workaround/approach seems to carry over fine for agents if accepted as it is now:

Really wishing for a centralized configuration store in some way but leaving that alone for now and moving forward toward three-build-or-higher integrated/distributed application with Meteor implementing pre-existing design and migrating over prior systems in play incrementally.

9

Seems with an opinionated stack I can spike out everything without blocking issues then circle back to general concepts / other or how to DRY up even more, and make agent implementations easy to add:

  • minio in common between client and server and agent via slingshot if not directly
  • Gitea container and package registry ( interchangeable, but focusing on independence by default )
  • meteor-base with startup.sh to pull settings.json into process.env.METEOR_SETTINGS
  • Dockerfile.* instances per flavor of server/agent side-by-side and build/deploy scripts
  • Meteor with mongo and hopefully imports/ that can be pulled into Agents not just Clients
  • Client with meteor-vite and vite-bundler packaging Vue.js and Tailwind
  • BullMQ with redis and not just streams of jobs, but beyond-cron job scheduling
  • Server with both BullMQ ( Queue abstracted ) and Meteor ( and WebApp abstracted )
  • Agent(s) with BullMQ ( Worker abstracted ) and just Mongo without Meteor
  • Native applications as variants of Client with the options there ( seeing discussion about changes to Electron and Cordova details, and similar, but have not gone native yet; would hold out a while )

No worthwhile framework for agents, but that is the most proprietary and “designer specific” aspect, and for me having the most to port from prior systems in other languages and modalities into this environment.

10

throw is the right move for many situations, versus try to recover, or after recovering is not possible… which is a major agent fundamental. “Explanation” is not the point, but identification by another agent or a monitoring system. You want something to die at the hands of the author versus circumstance, in a pre-sorted category of toe-tags, not just keep trying forever silently. In short, the “double tap” rule from Zombieland applies, but with the Idiocracy “scannable” convention.

That presents another gotcha hit quickly gotten through by opinionated abstraction to remove inside-baseball and go agnostic while staying DRY:

Meteor.Error requires Meteor like .settings and also radically limits granularity on meaning, relying on text and “message” versus “signal” of error with specificity on the face by object ( “medium is message” )

Abstracting that and ( carefully ) extending Error into classifications which then also need no message except for convenience in debugging. Easier to monitor and self-diagnose by typeof ( or ex.name ) down the line.

11

Next gotcha is asynchronous logging services shared by server and agents ( even more “isomorphic” than client and server together ) which seems like a dead/rough topic, and prone to paywalls / third-parties.

And from side angle

And

Or

( Updated 5-7 years ago except for FUNDING.md which is bad sign )

This is at design/convention level; am seeing 824 references to console.log in meteor/meteor :grimacing:

As I rtfm and search for "log" … :zero:

That gives me pause; one would think there is something important as non-blocking logging that can be syndicated rather than watching container log streams or local flat files. The mechanism of capturing the log itself is key. I think there were comments about this lately with a click-bait title but the topic itself was not entered. The topic take was “for debugging” but logging is >100x for telemetry and systems monitoring at high-resolution to be able to know what to do next, or detect issues before they become a problem

That can be a bit of a can of worms so also abstracting that and wrapping console.log at least to be able to do that non-caveman way as soon as possible.

12

This is a requiem for the 2-4 hours spent tracing a “race condition” only to discover await being added to JSON.parse causes the opposite behavior.

Upon looking back at “what does Meteor do” I originally imagined await … :roll_eyes: … and I even pasted the code above :rage: Just wow. Oh well, much more acclimated :laughing:

Have unified configuration layer working which goes between all, using the same convention, and abstracting out Meteor.settings across the board, other than within Meteor itself.