CONVERSATION: In place of LiveQuery, how should Meteor build webserver tier subscriptions (i.e. for GraphQL/Relay)?

I agree. The only thing is that they [Samza + Datomic] ARE the database. I do think Clojure + ClojureScript + Datomic + Om Next is appearing as the “one true way.” There’s nothing that comes close to the “observables all the way down” thing.

So unless you’re using Datomic the only option we have is the “big complex [subscription] ORM” approach. Or I guess we could be limited to manually optimizing certain queries or RethinkDB + Mongo/Oplog.

I think there is no other way but to catalog the matrix of possible queries + mutations for which subscriptions need to be maintained. Secondly, I think it’s possible and worthwhile. It’s a complex and time-consuming, which is why I’m bringing it up so we can get multiple minds on the problem.

1 Like

In addition to all of the above, it’s very surprising how little thought MDG put into this. They seemed to have ruled it out super quickly and jumped to the oplog idea. Nobody is denying it’s more performant than this route, but this route opens the doors to all routes, allowing you to optimize and get that performance via deeper database integrations for the that support some form of subscriptions.

So has nobody really thought about this, come on!!!

@msavin I remember a while back u brought this up too. Wanna discuss this with me. Somebody out there must have really thought about how we can monitor and broadcast subscriptions strictly on the webserver layer. @msavin how would you handle the removal of docs from a subscribed dataset? Let’s take that as an example and get a real conversation goin here. Somebody anybody!!! This is the core of Meteor’s server side reactive problem domain and nobody knows a damn thing!! Honestly it’s pathetic. @slava I’m sure u know something. This information can’t be siloed anymore–there should straight up be a doc on the meteor website describing the problem domain and its various tradeoffs. @sashko you must know something too, let’s have the conversation here, it’s far less helpful for us to wait weeks while MDG’s staff figures it out on their own. I disagree with u on the blaze 2 proposal hack pad being unhelpful–it was very helpful to me to quickly get an overview of the blaze 2 problem domain. Besides as were linked in that one thread there are many apps we can use for brainstorming and collective decision making. What you are working on privately may not be the right or best path. With less minds on it there is less opportunity to figure out the best approach. @sashko u complain about the noise on these forums as if we are idiots. Truth is you guys best be on a level above your users, being that u build that platform and we use it! So rather than keep forum threads superficial when it comes to how we will evolve Meteor, direct the conversation and don’t be afraid to shut down point of views you find unproductive. Be a firm “beneveloent dictator for life” like Rich Hickey does so well for Clojure. By the way @sashko u may not realize it, but u r the leader since @gschmidt won’t do it and u participate 20x more than anyone else at MDG. Does MDG realize that the most successful open source communities generally have outspoken leaders directing the conversation daily??

Real conversation abouthe evolution of Meteor needs to happen here. We aren’t idiots, not all of us at least. Developers with the expertise to talk about the matters will emerge and those without the skills will just listen. @sashko u don’t need to be afraid of the conversation becoming unproductive–u have it within ur means to insure it is productive, and u in fact glean some insights from it. I find it to be a great opportunity that u r both the forum leader and have been assigned by @gschmidt to address the most pressing need: our response to GraphQL. Use the forum to direct its implementation and the conversation around it.

So how are we gonna upgrade past our oplog approach??? I’ve laid out several strategies to specific challenges to set the frame for the conversation. @sashko do u have any specifics to add to the conversation or just sound bites?

Ps. I’m not trying to target u @sashko. I just think u aren’t aware of what roll ur in. Perhaps u don’t want it cuz it’s not what u signed up for when you joined MDG. We could only hope that if u made the choice to take on more responsibility, u would be rewarded for it somehow. Perhaps u feel u don’t have the sanction to fully share ur opinions and u feel ur more in the roll of a leveler–someone who levels the playing field and makes sure the conversation stays level headed. All I gotta say to that is its all a mistake and MDG needs someone leading with their point of view. U guys are all walking on egg shells.

The answer to the problem is simple: copy what Om Next is doing and evolve it in any directions they missed. One example of how we could improve it is combining routing into the equation just as redux+router solutions are emerging. State is being singularly managed by one large apparatus that addresses a bunch of concerns that all touch state in a factored solution. More and more aspects are being sucked into the orbit of this apparatus. The ones like routing which aren’t addressed remain a problem point and cause of impedance. All these new solutions have somehow managed to pinpoint long standing problems for which we previously accepted writing spaghetti to solve. Om Next for example solves the impedence of redux and relay both handling different types of state neither being good at both-- addresses handles domain state horribly and relay misses the mark for client UI state.

What I’m saying is: there is a precise vision for what we should do, the path has been laid out for us; surely it is a heavy duty undertaking; and before it goes anywhere we all gotta work together; this isn’t something package developers can address, Meteor currently isn’t open enough for that to be an option, therefore MDG has to address it–otherwise we wouldn’t need to have these conversations.

Oplog tailing in itself is not an inefficient approach and was the only logical choice considering MDG was using mongoDB already. However depending largely on how the data is consumed you could ruin this event based approach to live query. If you are talking about each node js process tailing the Oplog and maintaining state, then that is beyond insanity. This is how you have folks with merely 1k concurrent users spinning up 20 containers to support the traffic.

MDG is on the right track but it will require a proxy server not written in node js which act as a broker between the app containers and the underlying data source.

With this in mind, this proxy server will have to be made to work with mongodb first and then extended to other data sources. External data sources will have to trigger the proxy server with mongodb style oplog entries which is not science and very straight forward.

It sounds like u r sharing some very useful information, but it’s a bit high level. If you have the time I would love to hear a more detailed explanation of your response. I don’t fully understand the solution you’re mentioning. In addition if it can be extended to other databases, I’m not sure how it’s different from the solution this thread is trying to discover. Going to great lengths of writing proxy servers in, for example, Rust are not out of the picture; if that’s what must be done we should at least know it. At the end of the day it seems ur just suggesting the optimized version of exactly what I’m talking about but delegating to a centralized proxy server for filtering and broadcasting relevant mutations.

I’d love it if you presented a thorough breakdown for how the approach I’ve been laying out won’t work. After all, one of the initial premises of this conversation was discovering why it might not even be possible or worthwhile.

To build such a system, there has to be clear goals as to what is to be accomplished. Is MDG simply responsible for listening to changes in the app layer from a data source and then manipulating the DOM on the client or MDG is responsible for data transformations such as sorting and filtering. From what i have seen from MDG i believe what they are trying to accomplish is the latter.

The problem with performing this mutation in the app layer if we should forgo consistency in state for all connected clients (although that will be impossible in the financial world) is computational power. Currently MDG is computing these changes in each app container that is added to the cluster, so in actuality if a single container can support 10 concurrent users while the results of say a sort query is continuously been computed in the background, adding an extra container simply gives you another 10 more users because you are actually duplicating the same effort in the other container. To make the situation worse this is been done in javascript, creating objects left and right and relying on garbage collected V8 to bail you out. MDG however, i should say is very consciously aware of this limitation but decided to ship anyways. Watch one of their videos on Oplog tailing they are already aware of the limitations and even suggested a possible hack to the mongodb database to support live queries, although that is also a real bummer because we are trying to support multiple databases.

A standard proxy which listens for changes from multiple data sources in say the oplog format will solve the multi data source issue. For example if triggers is what is been used in postgres, a stored procedure can easily generate an oplog entry and send it to the proxy same as other databases. It also removes the duplication of effort issue.

The proxy will have to standardized in such a way that on updates to a row all required fields for a given publication are also published to the proxy server. This can be done because on startup meteor already has all the required meta data on all publications.

For your age and updated sort example. if a document is deleted, with the current approach All 20 running containers will have to issue an extra request back to the database to maintain the sort and limit state in memory. CPU burned in app container, CPU burned on the database. With a single proxy, we suffer the round trip once without having to do 20 sorts by all running app containers. Now i believe i saw a video you can search for it on youtube where one startup was running 60 containers to support live query. Yikes! A mess all over

1 Like

Excellent, this is exactly the sort of conversation I’m trying to get going.

…So basically what I’m hearing is that the logic I’m recommending we catalog is the right path, but it can’t be something that all servers in a cluster are doing. The conclusion is let’s take this same line of thinking and stuff it on another set of servers devoted to subscriptions + database stuff in between your typical webserver and the actual database.

This is likely similar in some ways to what Datomic is doing, but only part of the abstraction as well as being database-agnostic.

…Separately centralizing it in such a way still doesn’t seem like the hard part (i.e. communicating between servers)–actually implementing the various heuristics in a performant way still is. What other common heuristics do you see regarding the wide variety of queries and mutations that subscriptions must monitor?

1 Like

Also, check out this:

That seems to be exactly what you’re talking about. It’s brand new, I’m not sure how ready it is. It also offers a slightly different interface than what our usual Mongo cursors, but I wonder if it could be modified to monkey-patch the interface we are used to so we don’t have to change our code.

@artpolikarpov hey there, I don’t suppose you wanna join the conversation?

Also, Check this out…

https://nodechef.com/qtail

I think this solves the problem too. I don’t have to competency to evaluate it though. Maybe it can be added to this discussion.

3 Likes

this is speculation. You underestimate how much thought goes into designing these features and how much thought is put into the long term vs short term strategy and research.

A lot of people have thought about it. I wouldn’t talk for everyone, but I tend to not write walls of text unless I have something substantial to contribute/share.

Guess what? This is how people have been doing it forever. The problem is that it is hard to guarantee ordering/absence of concurrency issues and to make work with multiple writers. Eventually everything is “solved” by introducing message queues and/or redis/memcached. Nobody is keeping from going this route.

This is offensive to me and I think makes fun of everyone who has ever tried to solve hard problems and show their solutions to other people. Again, judging by your post history, you seem to “overshare” your thoughts without filtering, which results into unreadable walls of opinions, which have little technical value. If I were you, I would keep myself from calling people “pathetic” if you really want them to help you. Behavior like this is likely to stop me from looking at your posts in the future, as all I can see in them is disrespect to my time and disrespect to my work.

Go online, google search, watch youtube talks, oplog tailing wiki page, oplog tailing source code. Go read how Elixir/Phoenix/Derby/others do it. I read source code of similar projects often, I read how deepstream/socketstream/derby worked when I saw them on HN. How? I read the code. Read: I did my homework, instead of whining on the Internet begging others to spend their time on me. Show some work!

It is far less productive for anyone like Sashko to spend hours reading walls of criticism based on hype and opinions on twitter/hn, especially if it doesn’t provide any value. Without a question, Sashko and MDG gain a lot of prospective by reading useful posts, but taking a dump on forums in 20 threads with inflammatory titles, communicating the same point over and over again is not helpful to anyone.

See above.

I can’t believe you expect anyone to spend time thinking about your proposals if you can’t spend 2 minutes revising your post, typing in complete words and sentences and** NOT **FORMATTING your TEXT like YOU ARE on MYSPACE.

I disagree. Most successful opensource projects are driven by a lot of corporate development and strong organization without distractions on people who come and complain (see hadoop, chromium, linux kernel and python interpreter).

I think MDG had a history of looking at successful ideas and implementations and incorporating them in their projects. Arunoda had kinda working oplog-tailing, that MDG read inside out and took a lot of good ideas from in their implementation. When we were experimenting with SQL, we talked to Numtel and took his ideas and work on querying Postgres almost as is (with some refactorings and different code structure; the SQL queries are almost identical). Sashko looked at Mitar’s Blaze components when he was experimenting with Blaze 2 ideas. MDG looks at autoform for recommending the best forms package in the guide. Meteor’s React integration project researched every single Meteor/React integration available at the time.

Stop undermining work that goes into this project by spreading misinformation and creating an unhealthy atmosphere on forums.

Code and implementation speaks far better than “you guys should just do X. drop everything you are doing at the moment and listen to me”.

16 Likes

@slava first of all good to hear from and I’m finally glad to get a detailed response from someone affiliated with MDG. I’m gonna read all u wrote as soon as I get a chance (I’m out for the night), but I quickly got the tone. Don’t think for a second I don’t know how I am presenting myself. I’m presenting myself this way cuz there is a grievance here, beyond any one topic matter. Focusing on my tone is one way u could take it (which is very minor by the way), or u could look to the bigger picture I’m indicating. U don’t frequent these forums often anymore, so no real blame lies with u, but overall the expression of my grievances and what I’m trying to point out extends past this thread. So sorry to catch u in the mix of something that really isn’t ur fault or anyone person’s fault.

Overall the point is MDG–even tho u don’t work there anymore–doesn’t share much real info. Of course I know people worked on this before or the current inplementation wouldn’t exist. So it doesn’t make sense that somebody can’t step up and answer these questions. Getting short blurbs from u and @sashko is unnacceptable. It’s not just this thread but many others. The Meteor forum and the way MDG deals with the outside world leads to very little interaction by community members with what’s under the hood. The meteor framework is presented as a black box to everyone that doesn’t work at MDG. There is literally no point in community members contributing to code because its so tightly controlled by MDG, with all the sub packages locked inside one big repo even tho many could be broken out as their own repos on GitHub. This is a problem. The sharing of info about MDG’s plans to evolve meteor or rather lack thereof is an extension of that. Very little inside knowledge from MDG is contributed back this forum. Whatever was done the first time around could be very helpful if the problem domain was properly documented outside of code comments.

I’m gonna read the rest of what u got and I am really looking forward to actual info about the problem rather than just saying u don’t like the tone.

Again sorry for the tone. Not directed towards u. But hopefully u can see it’s a purposeful attempt to highlight a problem and get answers it. Judging by the length of ur response it looks like it worked.

And by the way, somebody should be able to spew out links that answer the questions posed if things are operating correctly. This is like the most important question about meteor and the feel of it is nobody really knows much. That is sad. This question is the backbone of everything related to GraphQL right now, it’s the backbone of everything meteor does server side and it’s the backbone of whether meteor will be able to survive as a platform. Tracker has already been deleted. Is how meteor handles reactivity server side also a second class solution. The lack of responses from anyone that still works at meteor would indicate that. That’s fine. Now it’s time for the community to help meteor because they clearly can’t do it on their own. If meteor doesn’t want to share info that’s fine. MDG employees hardly use this forum besides @sashko and apparently developers that don’t work there anymore. That’s a problem. As a long time meteor user, I’m very concerned about MDG’s behavior. It’s been a major disappointment. Other people wanna sugar code it and give u political bullshit, Slava–I’m telling u what it is brother. Meteor is on its way out if they don’t correct themselves. They got many things they gotta correct. Their galaxy success will be short lived if developers stop picking meteor for new projects. And for the first time ever there are better options than Meteor. Subjective? Predictions? Speculation? Nobody is gonna produce statistics! Get over it. But my speculations are hard to deny nevertheless. For the target audience meteor has traditionally focused on, mark my words, meteor is gonna fall out of favor. And quick. The more u or anyone defends them rather than face the true problem the more u r responsible for killing meteor. I used to recommend every day multiple times to use meteor. That’s not the case at this very moment. Though I’d like it to be the case again. So how do we rebuild this without mongo in a db-agnostic way geared towards a GraphQL/Relay future?

@slava ok, i read what you had to write. Here’s the thing, a lot of what you had to say is understandable. I’m gonna choose to not take offense and keep the discussion focused on actual issues. I hope you can also see how it’s understandable that I perceive Meteor to be in a very dangerous position having been beat out by competition for the first time in its history.

This isn’t about me and my ego, but I have put in lots of “work” to determine Meteor’s precise place within current trends. I may be bringing up a lot of the same points and perhaps you disagree with post length, but there’s a reason I’m getting emails and private messages and invitations to do interviews on the daily: because I’m giving a detailed voice to concerns that take a lot of research to truly understand that our community’s amazing developers don’t all have time to address themselves, which MDG wants to downplay.

The particular issue of researching how to build subscriptions isn’t something I perceive that I need to spend hours researching. The wheel shouldn’t have to be re-invented or re-researched. I generally perceive that behavior as young developers trying to boost their egos and show how smart they are, which I don’t find productive for me.

There’s a reason I opened this thread with a question: I don’t have the answer, I think those that do (clearly not you) could easily explain it, and I have reached a point in my career where I don’t have to do everything myself to feel good about myself. I don’t know the answer, I’m showing work in related areas to pin-point the trends, I’ll do the research if I have to, but I think it’s fair that I ask. And I do think the behavior shared by both MDG and the community of not knowing how our core subscriptions feature works and keeping the information siloed is “pathetic.” If you want to take one word, one sentence and interpret it to be an attack on you and other developers, that’s your choice. In this case, it’s hardly what you made out it to be. The golden rule (or rather, a golden rule) is: there is no offense unless it’s taken. You clearly invested time doing your research on what I have to say–what I have had to say is edgey, but ultimately makes important points, and at the end of the day it’s clear I’m not trying to make anyone feel bad, but just steer our focus to under-served areas. We are strong resourceful developers, and I don’t believe in singing kumbaya when the ship is burning. However, I do get it, when you worked at MDG they likely truly helped foster your growth and you’re sticking up for them. Meteor has problems @slava, and they seem to be starting to get addressed, but I’m gonna voice my concerns until I feel they have amply been addressed. MDG can kick me out if they choose, and that would be their loss.

So anyway, Meteor isn’t an open source project. This is a major problem because Meteor isn’t a single focused project but a conglomerate of many sub-projects. Projects that aren’t being attended to–just look at other threads at the top of the forum today. If I could easily fork and switch out the subscription stuff without resolving to hacks (like you can do in the rest of NPM) you wouldn’t be hearing my vocal side in this forum. Other developers have been content to monkey-patch everything. That’s their prerogative, but keep in mind that Meteor Hacks was named Meteor Hacks for a reason–literally every single package created under the Meteor Hacks brand was a monkey-patched hack. Everybody seemed to skip over that aspect. That’s worked out for @arunoda as after several years he’s been able to redirect the respect and fame toward Kadira, but that’s not a path I’ve found value for me to invest my time. Other languages don’t even have support for monkey-patching. That it boils down to that is one of the #1 reasons you shouldn’t use Meteor–it relies too heavily on hacks.

I think when this is addressed, a lot of problems will be solved. I recently myself have gotten tired of writing all this, and I’ve pin-pointed the root issue that would solve most everything to be: make Meteor sub-packages their own repos, and provide syntax in the packages file to refer to forked versions of core packages. The latter which is one of many ideas that nobody at MDG has cared to take the time responding to. Contrary to statements you made, there are open source communities where its core leaders communicate on the daily, giving good ideas some sort of acknowledgement. It’s clear Meteor is overworked and spread too thin, and that’s why it’s allowing competition from all sides to take its throne.

As far as the initial question that was asked, we need to figure out how we are going to bring subscriptions to the GraphQL future. If done truly right can continue to be the #1 attraction of Meteor. There needs to be a heavy duty undertaking, which likely only MDG can afford, to do this all over again and better than ever. The information I’ve been told is @gschmidt sent @sashko on some experimental scouting. That might already be too little too late.

Lastly, this thread is still open. Anyone with expertise in this area, it would be great to hear from you. I think Relay and Om Next have laid out a great foundation for a new full stack reactive interface, but my perspective is that what’s most important is working out the challenges regarding the db implementation since we wouldn’t be using a single db that has native support for subscriptions.

If you would like to contribute to this thread please keep it to discussing the problem domain, rather than bike shedding about tone and process etc. Lengthy responses welcome. Our community has a problem on our hands, let’s solve it together, and at the very least get informed once and for all about the deepest most important inner workings of our beloved platform.

I just want want to second @slava here. You’re very disrespectful @faceyspacey. This happened to me in a prior conversation with you as well. Its a waste of our time an energy to deal with your rants. Try to be more constructive and less abrasive. This thread is dead.

4 Likes

Second it all you want, it’s not going to make the problems go way. If I recall that thread you are mentioning was about where you brought up your otherwise awesome Elmish product in response to problems I was pointing out about Meteor. In turn what I pointed out was you’re not contributing or recognizing the problems of Meteor by glazing over them (and you did completely glaze over what I said) and saying basically just use other tools than Meteor’s core tools, while marketing your own stuff!

My perspective is if the response to every aspect of Meteor’s stack is to use something else, then we have killed Meteor.

My perspective is you wouldn’t be “wasting [your] time” responding to my threads if you didn’t find value in my points. You should have just responded last time. Anybody who knows me knows I’m a reasonable person able to have real respectful conversations, though heated when it matters. It’s a different style, not suited for all, but it’s a style too many developers are backing down from these days. Just check out what Cycle’s founder went through with the Facebook team yesterday:

I tend to take Andre’s side. Developers are turning to mush. May the best ideas win. That’s what software has always been about. Sure, there’s a limit, but I’ve hardly breached it. Speak up and debate the software instead of whining about your feelings. In addition when money’s involved because you have many client projects based on Meteor, it becomes a whole new ballgame. That’s not the case for everyone, and may not be the case for many of MDG’s employees.

In addition, I’m actually more moderate than you perceive. I’ve witnessed way more harsh devs than me. I’m hardly harsh, I’m minorly prodding, and I genuinely wanna discuss the topics I’m bringing up with anyone that also finds them important.

I don’t get the sense that many people have the information to be able to even address these problems. So what seems to be happening (just like all the attention MDG generally got from the Blaze debacle) is people’s fear from not knowing is coming out. You happen to know a shitload about the trends, as your Elmish product showcases. You also worked on the aforementioned github project above which allows you to trigger refreshes on subscriptions, so it’s a shame that you’re choosing this route instead of sharing information. You have valuable information Chet, share it!

1 Like

Hey, @faceyspacey I watched you several times you are saying we do monkey patched code. Yes we started like that and we moved away from it long time ago.

They are all not hacks, that’s how we extend stuff in JavaScript. Redux use something similar too, They call it store decorators. So, if you done something correctly and know what you are doing, it works properly.

We named “MeteorHacks” was not because of we monkey patch stuff. Hack is not monkey patching, it’s about doing experiments. If there was no such experiments(from us, MDG or someone else) Meteor lives inside a dark side.

We changed the name just because for claims like this.

Please stop calling us that. That’s wrong.

We are doers

10 Likes

I’m simply pointing @arunoda the name was ironic. Like you said, you started out monkey-patching. I know you’re “doers,” it’s very admirable, and I too often stand up for the importance of “hacks” and how the web has always been pushed forward by what was initially hacks. But it’s a sad state when you can only “do” in a select few areas, and in main very crucial areas there is little opportunity for the democracy of forks to have much meaning. A lot of problems have gone on too long. Your initial oplog tailing solution from long ago–and I don’t know what exactly went on there–could have been what itself became Meteor’s solution had it been more easy to merge in sub-packages. And you as a community member would likely still be leading it, rather than resources being centralized (and therefore limited) once again within Meteor. I’m sure you’ll find some way to refute my point by not looking past the specifics of that particular situation, but it’s the general implications that matter. Right now there is another fresh thread at the top of this forum about precisely this problem: open source contribution to Meteor isn’t happening.

Unbundling of Meteor is what we should be fighting for instead of the status quo, which is what I perceive you to be doing. What? In the name of friendly sound bites that keep less informed users as customers?? Breaking out sub-packages into their own repo is the tip of the ice berg.

Anyway, it’s a shame all your doing is only acquiring partial the value it could.

Guys, what’s going to happen here if we’re not careful is someone else outside of Meteor is going to create the standard for GraphQL subscriptions, and being that Meteor pioneered this, it shouldn’t be that way. The response may be: that’s fine, Meteor is the curator. That argument doesn’t stand when for an increasingly larger set of projects it’s better to forgo Meteor and just use plain NPM. Getting subscriptions right once again could be what leads to Meteor’s continued relevance. It’s how Meteor made its name; it would be a severe re-branding if that’s not the case. So I suggest we all talk about this once again.

I’m willing to watch my tone if we’re all willing to have honest and thorough discussions about our problems. “Speaking” or “voting” through the delivery of actual software (monkey-patched or not) doesn’t have to be the only way. That should be a fallback. A conversation by those with the expertise would save all parties time. How often @arunoda have you built stuff that is now irrelevant? I’m really asking that to all developers. I know I’ve been through this all too many times. It’s the nature of software, but we can do better if we agree ahead of time. I’m not in the business–not anymore at least–of building software people don’t want or agree to, especially in regards to a near-closed-source platform.

In this community, MDG has led the agreement of what our tools are too much so. That was awesome pre-galaxy, but at an all too pivotal time Meteor has found itself ill-prepared to embrace the knowledge of the community. I was relieved to hear in response to the “How Open Source Meteor really is…” thread, @sashko mentioned “We hope to soon move parts into their own repositories, so that it can be clear who owns what.” However, it’s still concerning that MDG’s default response is the first point @sashko made: “Not enough bandwidth - we are actively hiring, including for people that can help run a better community involvement program.” There isn’t sufficient community outreach when the default response is to hire more. Sure it all may lead to eventual better community participation, and it may very well be what’s needed. But this topic is being presented now–like many others–and the standard response is something to the tune of: we are working on it ourselves within MDG, don’t worry about it.

Large swaths of doers–your customers @arunoda–are gonna leave Meteor by 2017 if they can do better elsewhere as they can at this very moment in time. Let’s make this a short-lived blip and change the tides once again, and asap.

Hi, @faceyspacey - thanks for all of the thoughts! I haven’t had enough time to read all of this yet, but will definitely catch up ASAP.

You might be interested in our most recent blog post here: http://info.meteor.com/blog/reactive-graphql

2 Likes

I think the best answer is roughly:

  1. Make subscriptions optional. (As we’ve discussed doing in Reactive GraphQL.)
  2. Provide a scalable multi-process invalidation API that lets the application developer narrowly control recomputation from application server code.
  3. For the many cases where performance isn’t critical (eg. apps with a modest number of users, specific queries in large-scale apps that aren’t heavily subscribed to, prototypes), automatically invalidate queries on any write.
  4. Make poll-and-diff an option; only apps that have external database writers should need to turn it on. (In most Meteor apps it’s completely unnecessary to poll the DB on an interval timer in the absence of a recent DB write from a Meteor server.)

Once we have that, there’s a lot less pressure to build automatic, efficient invalidations.

4 Likes

For the multi process invalidation API, it’s great to allow many different solution as the communication platform. For an example:

  • Use redis
  • Use kafka
  • Use some network layer broadcasting mechanism (So, galaxy can do it automatically)

Also, it’s better to provide to configuration flexibility to client as well. So, in the Desktop app can re-fetch data for every few seconds.

In mobile, app can restrict realtime-ness.

1 Like

Yeah, I think we will probably just define an API between the application server and the invalidation server, and you can use whatever you want to build it, whether it’s redis, etcd, or kafka, or whatever. And then we will have an in-process one for development mode/reference that just runs inside your server.

Of course, Galaxy will run that part for you and it will be carefully optimized, but it is important that this part is well-documented so that you don’t have to use Galaxy if you can’t or really don’t want to for some reason.

I think this is why we want to invert the control so that the client is in charge of requesting new data and specifying what needs to be reactive rather than the server always pushing changes no matter what. We also hope that our architecture will allow a less stateful option that will be better for unreliable internet access, at the expense of higher latency. So you can pick if you want super low latency and high statefulness, or the other way around.

2 Likes

I’m basically gonna close this thread because I think my main question was answered by @sashko’s announcement. The question was whether it is possible to make a robust fine grained query invalidation engine/appliance/abstraction purely on the webserver layer (not so much stuff related to Redis and the like, which could be used to optimize it). Though we didn’t achieve the secondary aim of cataloging all such particular heuristics, I think we now have a platform to do so:

I’m interested to see how GraphQL types let’s us craft this abstraction. I can imagine a very design-patterny engine to synthesize the relation between GraphQL query types and mutation types to determine which queries need to be invalidated.

Ps. @sashko I like how the repo is called “data” by the way. That’s very bold! It very obviously indicates to me that what you are kicking off here is Meteor’s new abstraction when it comes to everything data-related, and your intention for it to ultimately replace a lot of old meteor machinery. It seems like the core basis for what Meteor 2.0 will be (in addition to the modules and hopefully Webpack-inspired work).

2 Likes