One good example of a corp owned open source project is Typescript.
They have a very clear policy about what PR they are accepting, and what the process is to propose new features.
I don’t know if they have a mega big team (actually they havn’t), but EVERY bug report and EVERY PR is getting a core team member (usually more) comment/flag within 12 hours on open days. I think it can be done when it’s really a goal of the project owners.
With Blaze being left out by MDG, letting the community to come up with PRs that would help in development of libraries such as Blaze Components or Manuel:Viewmodel while not breaking anything, would be a good move.
I just hope topics such as this get more visibility with MDG. @sashko this one seems like a no-brainer.
I’m looking at it this week. Like I said, we’re improving our internal organization and I now have agency over data-related components, so I can work on accepting this PR.
I’ve tried that before and usually get crickets in response… Maybe if the roadmap were more clear?
This doesn’t really address the core systemic problem of getting the community more involved in building Meteor itself. After looking over the top 20 projects on Github, it is clear to me that Meteor seems to ignore it’s community of potential contributors comapred to other projects.
Consider these numbers:
Now, many of you may be reading this and thinking that I am being biased in my picking (just sticking to Github top 20 that feel similar because they are programming frameworks). Let’s instead look at a top 20 project that is both open sourced and pursuing commercial interests: Docker.
In case this wasn’t enough to convince you that something is off, I decided to look at the number of stars and forks per project and come up with a stars to forks ratio (Forks / Stars = Ratio)
The interesting thing here is that Node actually moved it’s repo and the top 20 ‘archive’ repo actually shows a 23% fork to stars ratio. The jury is still out for me on what this signifies, but I think React is on the edge of having a healthly community of forkers, but it has a fast rise in Github stars. I think Angular is on the other extreme end of the scale with 46% ratio.
If you think about Node.js being the same ratio as Meteor, you have to also remember they went through significant growing pains last year when the community actually split in two due to stagnation of the framework and a lack of releases…
I point all of this out because I love using Meteor and want to see it stick around even if MDG doesn’t. Right now CollectionFS is being deprecated because a lack of community support and time needed to keep going. They have a 47 commit per contributor average. I just don’t see how Meteor itself is viable long term without more community engagement (accepted pull requests) on both sides here.
I have two theories about how this came about…
The first has to do with the MDG pedigree, which at one time was (and arguably still is) biased heavily towards MIT alumni. Its a major reason there has been enough confidence in Meteor for it to get to where it currently is. But there could be a not-invented-here attitude going on. I find this theory to be somewhat uncharitable however.
A more compassionate theory is simply that the testing infrastructure for the Meteor stack is more fragile than people realize; and there’s not enough visibility into the impact of changes that PRs will introduce. We’re just now getting test coverage across the entire Clinical release, and it’s already becoming quickly obvious that even the smallest changes can ripple across the entire distro.
On the other hand, Nightwatch/Gagarin has been handling some gnarly refactors of late. Stuff that I’ve been like ‘damn, did we just rip out that entire subsystem and replace it with no problem?’ So, I’m hopeful that with more QA visibility, there will be an increased acceptance of pull requests.
When we get the three reference apps all up and running; I’m considering giving a try at the Lodash refactor. We’re still on 1.1.0.3, so it would be an experiment at best. But it would be interesting to see the extent to which things break.
Glad to hear about the unbundling of meteor repos. That’s gone on too long. Now we need a mechanism to use a forked core package in our packages file.
These 2 changes will change everything for Meteor.
The problem with meteor vs say another tightly corporate-controlled project like Typescript is that Meteor’s scope is a lot broader and covers a bunch of tools that they aren’t individually deeply maintaining. So meteor could behave the way they do if they were just Typescript or Blaze or whatever, but in their case they have entombed a bunch of repos that they don’t have time for and that nobody else has the real option to contribute to.
It sounds like MDG is starting to wake up and smell the coffee. Hopefully it’s not too little too late. Where it currently stands, Meteor isn’t the best option like it was for several years, which disappoints me and more importantly is bad for my business. Doing something like Meatier on raw NPM where u can interchange any package is now the best option, whereas before the realtime stuff on the server didn’t compare. That is if plain JavaScript is a requirement–otherwise u have several more options.
A more compassionate theory is simply that the testing infrastructure for the Meteor stack is more fragile than people realize;
It is! Even worse, even tests which are currently there are failing by itself. It is really hard to know how pull requests influence things.
But most of my pull requests is exposing some internal symbols so that we can experiment with monkey-patching things and see what things need more public and stable API. Even that is too often too hard to get in. Like this pull request. There is no change to Meteor behavior. It just allows packages to do more.
And after the first feedback on the pull request, there was nothing back later on. And this is a problem. Because then I make an ugly workaround (often means cloning the whole Meteor into a subrepository in my package and simply overriding the whole thing). And then I am less interested in continuing working on the pull request. But this is a problem. Because somebody else who would like to do something similar might not have so much familiarity with Meteor code to know what exactly is there to change to allow them to make a feature they would like to see.
The first has to do with the MDG pedigree, which at one time was (and arguably still is) biased heavily towards MIT alumni. Its a major reason there has been enough confidence in Meteor for it to get to where it currently is. But there could be a not-invented-here attitude going on. I find this theory to be somewhat uncharitable however.
From my feeling of the code it is that it is of really amazing quality. I really loved that about Meteor. It is even bordering on over-engineered code. Code which really tries to assume all possible cases and solve all of them. This is really beautiful. But the issue is that it is clear that then some sub-optimal solutions are not really welcomed. Something which would be good enough for now, but not perfect long term. But it would help concrete people with concrete problems now. And then the answer is more like “your fix is not general enough, we will fix this someday, so do not worry”. But I would like a solution now and you can then generalize it when you have resources for that.
One issue I also think is the GitHub pull request model when one is dealing with so high-quality code project like Meteor. GitHub pull requests can be edited only by authors, not by the project. I think if GitHub would allow that MDG fixes some things directly in my pull request, it could make merging much faster. Then instead of making textual comments of what is necessary to be fixed for them to merge it, they could add also some changes directly to the code, I would check, modify further, and after few iterations it would be the code we would all agree on. Currently it is really strange. When I review my own pull requests it takes really a lot of time this back-and-forth when I could just make some few small commits, fix the styling stuff and so on directly. But I cannot. And for the project like Meteor I think this is even worse.
When I was working in the NY Presbyterian Healthcare System on Cerner EMR, we had more flowcharts than you could imagine.
Keep in mind that Cerner has 40,000 employees; and NYP had 10,000 employees. We ran QA tests 24/7, departments would queue up whose changes would get merged in what order; and system analysts were on emergency 24/7 call rotation to debug things when PR/upgrades inevitably broke things.
And it was the flow charts and architecture diagrams that kept everybody together on the same page. When things broke, we could trace the flow of the system, because the system was both documented and mandated to have QA test script coverage over all workflows.
My thinking is that architecture diagrams can communicate what Meteor is, how it’s architected, and help people communicate that “new feature Foo should be implemented here, not there” and “bug fix Bar changes this thing here, which will impact those downstream things there”. Architecture diagrams would give a lot of clarity on how the community can participate and contribute via pull requests.
I’m just not sure if we need the architecture diagrams to implement the tests; or the tests and use cases to document the architecture. There’s a chicken-and-egg aspect to it. Which comes first?
Yeah, I gave up trying to track down some code for autopublish, it was like 3 levels deep across packages and I gave up. I think you are right, The meteor package testing framework is fragile at best. I even tried to give MDG docs to add on how to properly run their test suite and I don’t think it was ever added to the wiki…
Yes, I think it’s clear that there are systematic problems that make it hard to both contribute to and accept contributions to Meteor, like the brittleness of tests, and lack of contribution guidance. In fact these are problems internally as well, when people join the company to work on Meteor it is hard to learn all of these things.
Unfortunately, this isn’t a problem that can be solved quickly - it has to be a turnaround in the culture that got us to where we are now. In my opinion, this means:
I think part of the first step is to have new projects start out as independent parts. A small nod towards this is the Meteor Guide repo, which is partially an intent to move the docs out of meteor/meteor, where people have found it very hard to contribute to the docs. But we can do much better and we are working on it.
I think once we split up some of the components, it will be simpler to give people in the community commit access to take a more active role in development, since it will be more clear what the APIs are, what the goal of that component is, etc.
This sounds great, is there anything we can do to help now?
I like what you’re saying; but I don’t think splitting the repository is a hard requirement for some sort of community governance / management of certain packages.
How about we establish a “Community Steering Committee”, to address the community’s pain points with core packages, but keep things in-line with MDG goals & prevent bloat/ feature creep.
The main work product of the committee could be a “Community Roadmap” which is essentially a list of well defined features/tasks that are “pull requests encouraged”, and can be reviewed & merged by community members.
I’d be happy to volunteer a few hours a week to make this happen.
@sashko Who would we need to talk to at MDG to get this officially sanctioned?
If anyone would like to get on board, please send me a message - our first goal is to produce a more refined proposal, (and then find the right ear).
You mean something like: https://github.com/MeteorCommunity/discussions/
We have done this quite some time ago. Feel free to see all tickets there. A lot of great people, a lot of great ideas. Not much was changed at MDG about this. Which is sad, because it is also visible that some active members there become through time less active in the Meteor community. This should be something alarming for Meteor.
To be fair, MDG was in the thick of Galaxy development a year ago. And there has been a minor amount of involvement from MDG already from Tim and Avital.
Now that Galaxy is out and attention is being turned back to the platform a bit, it would be great to see MDG simply fork the MeteorCommunity/discussions repo into meteor/discussions.
It’s also worth noting how wildly different the conversation is on MeteorCommunity/discussions than it is on the forums here. The general knowledge level of how Meteor works is much higher, and the Angular/React is a non-issue, whereas there is much more attention given to the rest of the stack.
I notice this as well, a lot of the posts I read were from prominent Meteor developers, I remember them as heavy hitters when I first came onboard over a year ago now. I don’t see them as frequently these days.
I saw @dandv on there, he was everywhere a year ago, I no longer see around much. He was organizing/building Meteor packaging for a while: https://github.com/MeteorPackaging
Of course React wasn’t really on the radar one year ago, at least in the Meteor community.
There is only as much energy one has to scream into the wall. Then you move on. Others do more or less forks (@awatson1978, @arunoda) and make their own things around and keep the head more or less down.
But there are also new people who come. But I worry that they will also burn out trying to achieve something and then leave again.
So the question is, what to do?
I also think that very poisonous was that there was for some time policy that there should be no feature requests on the issue tracker. This is also the reason why MeteorCommunity/discussions was made. To aggregate discussions about new features in one place. Now feature requests can be made on meteor core repository, but not much else changed about how much of those feature requests really become reality.
More than a year ago.