Also, check out this:
That seems to be exactly what you’re talking about. It’s brand new, I’m not sure how ready it is. It also offers a slightly different interface than what our usual Mongo cursors, but I wonder if it could be modified to monkey-patch the interface we are used to so we don’t have to change our code.
@artpolikarpov hey there, I don’t suppose you wanna join the conversation?
Also, Check this out…
I think this solves the problem too. I don’t have to competency to evaluate it though. Maybe it can be added to this discussion.
this is speculation. You underestimate how much thought goes into designing these features and how much thought is put into the long term vs short term strategy and research.
A lot of people have thought about it. I wouldn’t talk for everyone, but I tend to not write walls of text unless I have something substantial to contribute/share.
Guess what? This is how people have been doing it forever. The problem is that it is hard to guarantee ordering/absence of concurrency issues and to make work with multiple writers. Eventually everything is “solved” by introducing message queues and/or redis/memcached. Nobody is keeping from going this route.
This is offensive to me and I think makes fun of everyone who has ever tried to solve hard problems and show their solutions to other people. Again, judging by your post history, you seem to “overshare” your thoughts without filtering, which results into unreadable walls of opinions, which have little technical value. If I were you, I would keep myself from calling people “pathetic” if you really want them to help you. Behavior like this is likely to stop me from looking at your posts in the future, as all I can see in them is disrespect to my time and disrespect to my work.
Go online, google search, watch youtube talks, oplog tailing wiki page, oplog tailing source code. Go read how Elixir/Phoenix/Derby/others do it. I read source code of similar projects often, I read how deepstream/socketstream/derby worked when I saw them on HN. How? I read the code. Read: I did my homework, instead of whining on the Internet begging others to spend their time on me. Show some work!
It is far less productive for anyone like Sashko to spend hours reading walls of criticism based on hype and opinions on twitter/hn, especially if it doesn’t provide any value. Without a question, Sashko and MDG gain a lot of prospective by reading useful posts, but taking a dump on forums in 20 threads with inflammatory titles, communicating the same point over and over again is not helpful to anyone.
See above.
I can’t believe you expect anyone to spend time thinking about your proposals if you can’t spend 2 minutes revising your post, typing in complete words and sentences and** NOT **FORMATTING your TEXT like YOU ARE on MYSPACE.
I disagree. Most successful opensource projects are driven by a lot of corporate development and strong organization without distractions on people who come and complain (see hadoop, chromium, linux kernel and python interpreter).
I think MDG had a history of looking at successful ideas and implementations and incorporating them in their projects. Arunoda had kinda working oplog-tailing, that MDG read inside out and took a lot of good ideas from in their implementation. When we were experimenting with SQL, we talked to Numtel and took his ideas and work on querying Postgres almost as is (with some refactorings and different code structure; the SQL queries are almost identical). Sashko looked at Mitar’s Blaze components when he was experimenting with Blaze 2 ideas. MDG looks at autoform for recommending the best forms package in the guide. Meteor’s React integration project researched every single Meteor/React integration available at the time.
Stop undermining work that goes into this project by spreading misinformation and creating an unhealthy atmosphere on forums.
Code and implementation speaks far better than “you guys should just do X. drop everything you are doing at the moment and listen to me”.
@slava first of all good to hear from and I’m finally glad to get a detailed response from someone affiliated with MDG. I’m gonna read all u wrote as soon as I get a chance (I’m out for the night), but I quickly got the tone. Don’t think for a second I don’t know how I am presenting myself. I’m presenting myself this way cuz there is a grievance here, beyond any one topic matter. Focusing on my tone is one way u could take it (which is very minor by the way), or u could look to the bigger picture I’m indicating. U don’t frequent these forums often anymore, so no real blame lies with u, but overall the expression of my grievances and what I’m trying to point out extends past this thread. So sorry to catch u in the mix of something that really isn’t ur fault or anyone person’s fault.
Overall the point is MDG–even tho u don’t work there anymore–doesn’t share much real info. Of course I know people worked on this before or the current inplementation wouldn’t exist. So it doesn’t make sense that somebody can’t step up and answer these questions. Getting short blurbs from u and @sashko is unnacceptable. It’s not just this thread but many others. The Meteor forum and the way MDG deals with the outside world leads to very little interaction by community members with what’s under the hood. The meteor framework is presented as a black box to everyone that doesn’t work at MDG. There is literally no point in community members contributing to code because its so tightly controlled by MDG, with all the sub packages locked inside one big repo even tho many could be broken out as their own repos on GitHub. This is a problem. The sharing of info about MDG’s plans to evolve meteor or rather lack thereof is an extension of that. Very little inside knowledge from MDG is contributed back this forum. Whatever was done the first time around could be very helpful if the problem domain was properly documented outside of code comments.
I’m gonna read the rest of what u got and I am really looking forward to actual info about the problem rather than just saying u don’t like the tone.
Again sorry for the tone. Not directed towards u. But hopefully u can see it’s a purposeful attempt to highlight a problem and get answers it. Judging by the length of ur response it looks like it worked.
And by the way, somebody should be able to spew out links that answer the questions posed if things are operating correctly. This is like the most important question about meteor and the feel of it is nobody really knows much. That is sad. This question is the backbone of everything related to GraphQL right now, it’s the backbone of everything meteor does server side and it’s the backbone of whether meteor will be able to survive as a platform. Tracker has already been deleted. Is how meteor handles reactivity server side also a second class solution. The lack of responses from anyone that still works at meteor would indicate that. That’s fine. Now it’s time for the community to help meteor because they clearly can’t do it on their own. If meteor doesn’t want to share info that’s fine. MDG employees hardly use this forum besides @sashko and apparently developers that don’t work there anymore. That’s a problem. As a long time meteor user, I’m very concerned about MDG’s behavior. It’s been a major disappointment. Other people wanna sugar code it and give u political bullshit, Slava–I’m telling u what it is brother. Meteor is on its way out if they don’t correct themselves. They got many things they gotta correct. Their galaxy success will be short lived if developers stop picking meteor for new projects. And for the first time ever there are better options than Meteor. Subjective? Predictions? Speculation? Nobody is gonna produce statistics! Get over it. But my speculations are hard to deny nevertheless. For the target audience meteor has traditionally focused on, mark my words, meteor is gonna fall out of favor. And quick. The more u or anyone defends them rather than face the true problem the more u r responsible for killing meteor. I used to recommend every day multiple times to use meteor. That’s not the case at this very moment. Though I’d like it to be the case again. So how do we rebuild this without mongo in a db-agnostic way geared towards a GraphQL/Relay future?
@slava ok, i read what you had to write. Here’s the thing, a lot of what you had to say is understandable. I’m gonna choose to not take offense and keep the discussion focused on actual issues. I hope you can also see how it’s understandable that I perceive Meteor to be in a very dangerous position having been beat out by competition for the first time in its history.
This isn’t about me and my ego, but I have put in lots of “work” to determine Meteor’s precise place within current trends. I may be bringing up a lot of the same points and perhaps you disagree with post length, but there’s a reason I’m getting emails and private messages and invitations to do interviews on the daily: because I’m giving a detailed voice to concerns that take a lot of research to truly understand that our community’s amazing developers don’t all have time to address themselves, which MDG wants to downplay.
The particular issue of researching how to build subscriptions isn’t something I perceive that I need to spend hours researching. The wheel shouldn’t have to be re-invented or re-researched. I generally perceive that behavior as young developers trying to boost their egos and show how smart they are, which I don’t find productive for me.
There’s a reason I opened this thread with a question: I don’t have the answer, I think those that do (clearly not you) could easily explain it, and I have reached a point in my career where I don’t have to do everything myself to feel good about myself. I don’t know the answer, I’m showing work in related areas to pin-point the trends, I’ll do the research if I have to, but I think it’s fair that I ask. And I do think the behavior shared by both MDG and the community of not knowing how our core subscriptions feature works and keeping the information siloed is “pathetic.” If you want to take one word, one sentence and interpret it to be an attack on you and other developers, that’s your choice. In this case, it’s hardly what you made out it to be. The golden rule (or rather, a golden rule) is: there is no offense unless it’s taken. You clearly invested time doing your research on what I have to say–what I have had to say is edgey, but ultimately makes important points, and at the end of the day it’s clear I’m not trying to make anyone feel bad, but just steer our focus to under-served areas. We are strong resourceful developers, and I don’t believe in singing kumbaya when the ship is burning. However, I do get it, when you worked at MDG they likely truly helped foster your growth and you’re sticking up for them. Meteor has problems @slava, and they seem to be starting to get addressed, but I’m gonna voice my concerns until I feel they have amply been addressed. MDG can kick me out if they choose, and that would be their loss.
So anyway, Meteor isn’t an open source project. This is a major problem because Meteor isn’t a single focused project but a conglomerate of many sub-projects. Projects that aren’t being attended to–just look at other threads at the top of the forum today. If I could easily fork and switch out the subscription stuff without resolving to hacks (like you can do in the rest of NPM) you wouldn’t be hearing my vocal side in this forum. Other developers have been content to monkey-patch everything. That’s their prerogative, but keep in mind that Meteor Hacks was named Meteor Hacks for a reason–literally every single package created under the Meteor Hacks brand was a monkey-patched hack. Everybody seemed to skip over that aspect. That’s worked out for @arunoda as after several years he’s been able to redirect the respect and fame toward Kadira, but that’s not a path I’ve found value for me to invest my time. Other languages don’t even have support for monkey-patching. That it boils down to that is one of the #1 reasons you shouldn’t use Meteor–it relies too heavily on hacks.
I think when this is addressed, a lot of problems will be solved. I recently myself have gotten tired of writing all this, and I’ve pin-pointed the root issue that would solve most everything to be: make Meteor sub-packages their own repos, and provide syntax in the packages file to refer to forked versions of core packages. The latter which is one of many ideas that nobody at MDG has cared to take the time responding to. Contrary to statements you made, there are open source communities where its core leaders communicate on the daily, giving good ideas some sort of acknowledgement. It’s clear Meteor is overworked and spread too thin, and that’s why it’s allowing competition from all sides to take its throne.
As far as the initial question that was asked, we need to figure out how we are going to bring subscriptions to the GraphQL future. If done truly right can continue to be the #1 attraction of Meteor. There needs to be a heavy duty undertaking, which likely only MDG can afford, to do this all over again and better than ever. The information I’ve been told is @gschmidt sent @sashko on some experimental scouting. That might already be too little too late.
Lastly, this thread is still open. Anyone with expertise in this area, it would be great to hear from you. I think Relay and Om Next have laid out a great foundation for a new full stack reactive interface, but my perspective is that what’s most important is working out the challenges regarding the db implementation since we wouldn’t be using a single db that has native support for subscriptions.
If you would like to contribute to this thread please keep it to discussing the problem domain, rather than bike shedding about tone and process etc. Lengthy responses welcome. Our community has a problem on our hands, let’s solve it together, and at the very least get informed once and for all about the deepest most important inner workings of our beloved platform.
I just want want to second @slava here. You’re very disrespectful @faceyspacey. This happened to me in a prior conversation with you as well. Its a waste of our time an energy to deal with your rants. Try to be more constructive and less abrasive. This thread is dead.
Second it all you want, it’s not going to make the problems go way. If I recall that thread you are mentioning was about where you brought up your otherwise awesome Elmish product in response to problems I was pointing out about Meteor. In turn what I pointed out was you’re not contributing or recognizing the problems of Meteor by glazing over them (and you did completely glaze over what I said) and saying basically just use other tools than Meteor’s core tools, while marketing your own stuff!
My perspective is if the response to every aspect of Meteor’s stack is to use something else, then we have killed Meteor.
My perspective is you wouldn’t be “wasting [your] time” responding to my threads if you didn’t find value in my points. You should have just responded last time. Anybody who knows me knows I’m a reasonable person able to have real respectful conversations, though heated when it matters. It’s a different style, not suited for all, but it’s a style too many developers are backing down from these days. Just check out what Cycle’s founder went through with the Facebook team yesterday:
I tend to take Andre’s side. Developers are turning to mush. May the best ideas win. That’s what software has always been about. Sure, there’s a limit, but I’ve hardly breached it. Speak up and debate the software instead of whining about your feelings. In addition when money’s involved because you have many client projects based on Meteor, it becomes a whole new ballgame. That’s not the case for everyone, and may not be the case for many of MDG’s employees.
In addition, I’m actually more moderate than you perceive. I’ve witnessed way more harsh devs than me. I’m hardly harsh, I’m minorly prodding, and I genuinely wanna discuss the topics I’m bringing up with anyone that also finds them important.
I don’t get the sense that many people have the information to be able to even address these problems. So what seems to be happening (just like all the attention MDG generally got from the Blaze debacle) is people’s fear from not knowing is coming out. You happen to know a shitload about the trends, as your Elmish product showcases. You also worked on the aforementioned github project above which allows you to trigger refreshes on subscriptions, so it’s a shame that you’re choosing this route instead of sharing information. You have valuable information Chet, share it!
Hey, @faceyspacey I watched you several times you are saying we do monkey patched code. Yes we started like that and we moved away from it long time ago.
They are all not hacks, that’s how we extend stuff in JavaScript. Redux use something similar too, They call it store decorators. So, if you done something correctly and know what you are doing, it works properly.
We named “MeteorHacks” was not because of we monkey patch stuff. Hack is not monkey patching, it’s about doing experiments. If there was no such experiments(from us, MDG or someone else) Meteor lives inside a dark side.
We changed the name just because for claims like this.
Please stop calling us that. That’s wrong.
We are doers
I’m simply pointing @arunoda the name was ironic. Like you said, you started out monkey-patching. I know you’re “doers,” it’s very admirable, and I too often stand up for the importance of “hacks” and how the web has always been pushed forward by what was initially hacks. But it’s a sad state when you can only “do” in a select few areas, and in main very crucial areas there is little opportunity for the democracy of forks to have much meaning. A lot of problems have gone on too long. Your initial oplog tailing solution from long ago–and I don’t know what exactly went on there–could have been what itself became Meteor’s solution had it been more easy to merge in sub-packages. And you as a community member would likely still be leading it, rather than resources being centralized (and therefore limited) once again within Meteor. I’m sure you’ll find some way to refute my point by not looking past the specifics of that particular situation, but it’s the general implications that matter. Right now there is another fresh thread at the top of this forum about precisely this problem: open source contribution to Meteor isn’t happening.
Unbundling of Meteor is what we should be fighting for instead of the status quo, which is what I perceive you to be doing. What? In the name of friendly sound bites that keep less informed users as customers?? Breaking out sub-packages into their own repo is the tip of the ice berg.
Anyway, it’s a shame all your doing is only acquiring partial the value it could.
Guys, what’s going to happen here if we’re not careful is someone else outside of Meteor is going to create the standard for GraphQL subscriptions, and being that Meteor pioneered this, it shouldn’t be that way. The response may be: that’s fine, Meteor is the curator. That argument doesn’t stand when for an increasingly larger set of projects it’s better to forgo Meteor and just use plain NPM. Getting subscriptions right once again could be what leads to Meteor’s continued relevance. It’s how Meteor made its name; it would be a severe re-branding if that’s not the case. So I suggest we all talk about this once again.
I’m willing to watch my tone if we’re all willing to have honest and thorough discussions about our problems. “Speaking” or “voting” through the delivery of actual software (monkey-patched or not) doesn’t have to be the only way. That should be a fallback. A conversation by those with the expertise would save all parties time. How often @arunoda have you built stuff that is now irrelevant? I’m really asking that to all developers. I know I’ve been through this all too many times. It’s the nature of software, but we can do better if we agree ahead of time. I’m not in the business–not anymore at least–of building software people don’t want or agree to, especially in regards to a near-closed-source platform.
In this community, MDG has led the agreement of what our tools are too much so. That was awesome pre-galaxy, but at an all too pivotal time Meteor has found itself ill-prepared to embrace the knowledge of the community. I was relieved to hear in response to the “How Open Source Meteor really is…” thread, @sashko mentioned “We hope to soon move parts into their own repositories, so that it can be clear who owns what.” However, it’s still concerning that MDG’s default response is the first point @sashko made: “Not enough bandwidth - we are actively hiring, including for people that can help run a better community involvement program.” There isn’t sufficient community outreach when the default response is to hire more. Sure it all may lead to eventual better community participation, and it may very well be what’s needed. But this topic is being presented now–like many others–and the standard response is something to the tune of: we are working on it ourselves within MDG, don’t worry about it.
Large swaths of doers–your customers @arunoda–are gonna leave Meteor by 2017 if they can do better elsewhere as they can at this very moment in time. Let’s make this a short-lived blip and change the tides once again, and asap.
Hi, @faceyspacey - thanks for all of the thoughts! I haven’t had enough time to read all of this yet, but will definitely catch up ASAP.
You might be interested in our most recent blog post here: http://info.meteor.com/blog/reactive-graphql
I think the best answer is roughly:
Once we have that, there’s a lot less pressure to build automatic, efficient invalidations.
For the multi process invalidation API, it’s great to allow many different solution as the communication platform. For an example:
Also, it’s better to provide to configuration flexibility to client as well. So, in the Desktop app can re-fetch data for every few seconds.
In mobile, app can restrict realtime-ness.
Yeah, I think we will probably just define an API between the application server and the invalidation server, and you can use whatever you want to build it, whether it’s redis, etcd, or kafka, or whatever. And then we will have an in-process one for development mode/reference that just runs inside your server.
Of course, Galaxy will run that part for you and it will be carefully optimized, but it is important that this part is well-documented so that you don’t have to use Galaxy if you can’t or really don’t want to for some reason.
I think this is why we want to invert the control so that the client is in charge of requesting new data and specifying what needs to be reactive rather than the server always pushing changes no matter what. We also hope that our architecture will allow a less stateful option that will be better for unreliable internet access, at the expense of higher latency. So you can pick if you want super low latency and high statefulness, or the other way around.
I’m basically gonna close this thread because I think my main question was answered by @sashko’s announcement. The question was whether it is possible to make a robust fine grained query invalidation engine/appliance/abstraction purely on the webserver layer (not so much stuff related to Redis and the like, which could be used to optimize it). Though we didn’t achieve the secondary aim of cataloging all such particular heuristics, I think we now have a platform to do so:
I’m interested to see how GraphQL types let’s us craft this abstraction. I can imagine a very design-patterny engine to synthesize the relation between GraphQL query types and mutation types to determine which queries need to be invalidated.
Ps. @sashko I like how the repo is called “data” by the way. That’s very bold! It very obviously indicates to me that what you are kicking off here is Meteor’s new abstraction when it comes to everything data-related, and your intention for it to ultimately replace a lot of old meteor machinery. It seems like the core basis for what Meteor 2.0 will be (in addition to the modules and hopefully Webpack-inspired work).
Thanks for the vote of confidence!
@faceyspacey I appreciate what you bring to the conversation. I think it’s wrong that some people are being judgemental and taking your writing style personally when you are obviously spending your unpaid time and efforts to get to the bottom of things. I like reading through the possibilities you outline, and how you generally conclude with your strongest argument. I do think though that more people would actually read your thoughts if they were indeed more condensed. But who cares! We aren’t paying you to write for us!
I also want to add that @sashko you always handle things so well and are a great presence on these forums.
I was wondering if it is possible to use event hooks in order to update data. In Postgres, there is event triggers that will fire. Would this make it possible to integrate postgres?
What if this was used to find all connected clients to this document and broadcast the change?
For me it’s amazing how differently something reads after you’ve heard the author talk. Have a listen of this to hear just how enthusiastic @faceyspacey is about all of this: http://www.meteorinterviews.com/13
Just wondering: what do you think about deepstream?
I haven’t really looked into it, but I like the way you can plug in different storing and caching layers.
I don’t remember much now, but at the time I looked at it, it was relying on Redis to have any real-timeness. I didn’t notice anything different from what socket.io+Redis already do. Here is my HN comment from the thread: https://news.ycombinator.com/item?id=9703552