Ouch - if people like @mitar have problems getting pull requests in, than there is definitely a problem. I’m sure this boils down to MDG not having enough time to properly handle the amount of pull requests they get, but regardless MDG please take a serious look at this. We really don’t want to lose potentially great contributions, especially from well respected community members.
MDG: I know it’s very time consuming to deal with pull requests and the necessary split up back and forth that they require, but maybe it’s time to involve a handful of top tier community members as pull request reviewers. Maybe the community can help offload some of your work? Trust us - we want you guys to focus on making big changes to the platform, and we want Galaxy to succeed. It’s just that we also want to make sure that the smaller (often quite impactful) stuff isn’t overlooked in the process.
Right now, it’s very hard to consider and accept pull requests for a couple reasons, most of which we are actively working on:
Not enough bandwidth - we are actively hiring, including for people that can help run a better community involvement program.
Too much code in one place - the Meteor repository is so monolithic it’s hard to run tests scoped to one thing, and nobody is really “in charge” of the whole thing. We hope to soon move parts into their own repositories, so that it can be clear who owns what.
Team structure - we’ve very recently organized into development teams that own specific parts of Meteor (literally last week), and hopefully this will give people the right autonomy to accept PRs into their own project, and if they don’t get accepted at least you know who to bother.
Also, I would say it is important to get consensus from people that a change is important before submitting a PR, so that there are no surprises. As they are now, a lot of PRs are actually feature requests or bug reports in disguise that just happen to come with some code.
Wow, this is really exciting to hear! So based on your contributions here, on the guide and on github @sashko, they’ve obviously put you in charge of, well, everything!
I am doing the Guide and spinning up on the new Data Team, which is going to be all about databases, data loading, accounts, client-side state, and related topics.
One solution to this is to allow community members to also commit to the repository. There could be clear rules: changes have to contain documentation, tests, there should not be any backwards incompatible changes. If there are, there should be some other clear escalation process and then it would be discussed internally at your meetings and you would get back with that.
Also, I would say it is important to get consensus from people that a change is important before
submitting a PR, so that there are no surprises. As they are now, a lot
of PRs are actually feature requests or bug reports in disguise that
just happen to come with some code.
But if the code is working, with tests, and documentation, and does not break anything for anybody else, but does solve a problem for a particular user or users?
And what about clear problems and solutions community already build pull requests for?
Since March 2013 a known problem. With @arunoda making a fix in 2014. And that was even discussed by core MDG members at the time when they were still active on tickets. And after that silence. And then 16 days ago asking @arunoda to do again some extra work to get this in, so he has to get his mind all again to it. And this is an example of a ticket with very high input from MDG.
Another example is https://github.com/meteor/meteor/pull/5680, where we get 6s -> 0.4s improvement in data processing. Which I think should be clear for anyone in Meteor world that it is a great improvement and should be released to the world as soon as possible. No more freezing UIs. I would even go and say that this is the main reason why people experienced sluggish Blaze.
Those are just two examples with which I have been involved with.
One good example of a corp owned open source project is Typescript.
They have a very clear policy about what PR they are accepting, and what the process is to propose new features.
I don’t know if they have a mega big team (actually they havn’t), but EVERY bug report and EVERY PR is getting a core team member (usually more) comment/flag within 12 hours on open days. I think it can be done when it’s really a goal of the project owners.
With Blaze being left out by MDG, letting the community to come up with PRs that would help in development of libraries such as Blaze Components or Manuel:Viewmodel while not breaking anything, would be a good move.
I’m looking at it this week. Like I said, we’re improving our internal organization and I now have agency over data-related components, so I can work on accepting this PR.
This doesn’t really address the core systemic problem of getting the community more involved in building Meteor itself. After looking over the top 20 projects on Github, it is clear to me that Meteor seems to ignore it’s community of potential contributors comapred to other projects.
Consider these numbers:
Meteor: 15,721 commits and 258 contributors = 60 commits per contributor on average. The first “non-MDG employee” contributor with commits is @mitar with 52. A long tail of 241 contributors fall below the average.
Rails: 55.954 commits and 2,963 contributors = 18.8 commits per contributor on average. The first non-core commit rights contributor starts somewhere around 200 commits, well above the average. You also have 3 guys at the top driving that average WAY up at 3k commits each. We cannot see the size of the long tail of contributors below the average because Github doesn’t show that many.
Node.js: 12,978 commits and 825 contributors = 15.7 commits per contributor on average. 60 contributors are above the average, leaving 765 contributors in the below average long tail.
React: 6,081 commits and 602 contributors = 10.1 commits per contributor on average. There are 32 commits above the average out of 602, so the long tail of help looks nice and healthy at 570 contributors below the average.
Angular: 7,428 commits and 1,378 contributors = 5.3 commits per contributor on average. There are 63 contributors who are above the average, so again a nice big healthy long tail of contributors at 1,315.
Now, many of you may be reading this and thinking that I am being biased in my picking (just sticking to Github top 20 that feel similar because they are programming frameworks). Let’s instead look at a top 20 project that is both open sourced and pursuing commercial interests: Docker.
Docker: 21,594 commits and 1,266 contributors = 17 commits per contributor on average. There are 87 contributors who are above the average, leaving 1,179 contributors in our below average long tail.
In case this wasn’t enough to convince you that something is off, I decided to look at the number of stars and forks per project and come up with a stars to forks ratio (Forks / Stars = Ratio)
Meteor - (3,742 forks / 31,556 stars) = 12%
Rails - (11,905 forks / 29,349 stars) = 41%
Node.js - (2,359 forks / 19,607 stars) = 12%
React - (5,735 forks / 35,349 stars) = 16%
Angular - (21,508 forks / 46,372 stars) = 46%
Docker - (7,750 forks / 28,040 stars) = 28%
The interesting thing here is that Node actually moved it’s repo and the top 20 ‘archive’ repo actually shows a 23% fork to stars ratio. The jury is still out for me on what this signifies, but I think React is on the edge of having a healthly community of forkers, but it has a fast rise in Github stars. I think Angular is on the other extreme end of the scale with 46% ratio.
If you think about Node.js being the same ratio as Meteor, you have to also remember they went through significant growing pains last year when the community actually split in two due to stagnation of the framework and a lack of releases…
I point all of this out because I love using Meteor and want to see it stick around even if MDG doesn’t. Right now CollectionFS is being deprecated because a lack of community support and time needed to keep going. They have a 47 commit per contributor average. I just don’t see how Meteor itself is viable long term without more community engagement (accepted pull requests) on both sides here.
The first has to do with the MDG pedigree, which at one time was (and arguably still is) biased heavily towards MIT alumni. Its a major reason there has been enough confidence in Meteor for it to get to where it currently is. But there could be a not-invented-here attitude going on. I find this theory to be somewhat uncharitable however.
A more compassionate theory is simply that the testing infrastructure for the Meteor stack is more fragile than people realize; and there’s not enough visibility into the impact of changes that PRs will introduce. We’re just now getting test coverage across the entire Clinical release, and it’s already becoming quickly obvious that even the smallest changes can ripple across the entire distro.
On the other hand, Nightwatch/Gagarin has been handling some gnarly refactors of late. Stuff that I’ve been like ‘damn, did we just rip out that entire subsystem and replace it with no problem?’ So, I’m hopeful that with more QA visibility, there will be an increased acceptance of pull requests.
When we get the three reference apps all up and running; I’m considering giving a try at the Lodash refactor. We’re still on 1.1.0.3, so it would be an experiment at best. But it would be interesting to see the extent to which things break.
Glad to hear about the unbundling of meteor repos. That’s gone on too long. Now we need a mechanism to use a forked core package in our packages file.
These 2 changes will change everything for Meteor.
The problem with meteor vs say another tightly corporate-controlled project like Typescript is that Meteor’s scope is a lot broader and covers a bunch of tools that they aren’t individually deeply maintaining. So meteor could behave the way they do if they were just Typescript or Blaze or whatever, but in their case they have entombed a bunch of repos that they don’t have time for and that nobody else has the real option to contribute to.
It sounds like MDG is starting to wake up and smell the coffee. Hopefully it’s not too little too late. Where it currently stands, Meteor isn’t the best option like it was for several years, which disappoints me and more importantly is bad for my business. Doing something like Meatier on raw NPM where u can interchange any package is now the best option, whereas before the realtime stuff on the server didn’t compare. That is if plain JavaScript is a requirement–otherwise u have several more options.
A more compassionate theory is simply that the testing infrastructure for the Meteor stack is more fragile than people realize;
It is! Even worse, even tests which are currently there are failing by itself. It is really hard to know how pull requests influence things.
But most of my pull requests is exposing some internal symbols so that we can experiment with monkey-patching things and see what things need more public and stable API. Even that is too often too hard to get in. Like this pull request. There is no change to Meteor behavior. It just allows packages to do more.
And after the first feedback on the pull request, there was nothing back later on. And this is a problem. Because then I make an ugly workaround (often means cloning the whole Meteor into a subrepository in my package and simply overriding the whole thing). And then I am less interested in continuing working on the pull request. But this is a problem. Because somebody else who would like to do something similar might not have so much familiarity with Meteor code to know what exactly is there to change to allow them to make a feature they would like to see.
The first has to do with the MDG pedigree, which at one time was (and arguably still is) biased heavily towards MIT alumni. Its a major reason there has been enough confidence in Meteor for it to get to where it currently is. But there could be a not-invented-here attitude going on. I find this theory to be somewhat uncharitable however.
From my feeling of the code it is that it is of really amazing quality. I really loved that about Meteor. It is even bordering on over-engineered code. Code which really tries to assume all possible cases and solve all of them. This is really beautiful. But the issue is that it is clear that then some sub-optimal solutions are not really welcomed. Something which would be good enough for now, but not perfect long term. But it would help concrete people with concrete problems now. And then the answer is more like “your fix is not general enough, we will fix this someday, so do not worry”. But I would like a solution now and you can then generalize it when you have resources for that.
One issue I also think is the GitHub pull request model when one is dealing with so high-quality code project like Meteor. GitHub pull requests can be edited only by authors, not by the project. I think if GitHub would allow that MDG fixes some things directly in my pull request, it could make merging much faster. Then instead of making textual comments of what is necessary to be fixed for them to merge it, they could add also some changes directly to the code, I would check, modify further, and after few iterations it would be the code we would all agree on. Currently it is really strange. When I review my own pull requests it takes really a lot of time this back-and-forth when I could just make some few small commits, fix the styling stuff and so on directly. But I cannot. And for the project like Meteor I think this is even worse.
When I was working in the NY Presbyterian Healthcare System on Cerner EMR, we had more flowcharts than you could imagine.
Keep in mind that Cerner has 40,000 employees; and NYP had 10,000 employees. We ran QA tests 24/7, departments would queue up whose changes would get merged in what order; and system analysts were on emergency 24/7 call rotation to debug things when PR/upgrades inevitably broke things.
And it was the flow charts and architecture diagrams that kept everybody together on the same page. When things broke, we could trace the flow of the system, because the system was both documented and mandated to have QA test script coverage over all workflows.
My thinking is that architecture diagrams can communicate what Meteor is, how it’s architected, and help people communicate that “new feature Foo should be implemented here, not there” and “bug fix Bar changes this thing here, which will impact those downstream things there”. Architecture diagrams would give a lot of clarity on how the community can participate and contribute via pull requests.
I’m just not sure if we need the architecture diagrams to implement the tests; or the tests and use cases to document the architecture. There’s a chicken-and-egg aspect to it. Which comes first?
Yeah, I gave up trying to track down some code for autopublish, it was like 3 levels deep across packages and I gave up. I think you are right, The meteor package testing framework is fragile at best. I even tried to give MDG docs to add on how to properly run their test suite and I don’t think it was ever added to the wiki…
Yes, I think it’s clear that there are systematic problems that make it hard to both contribute to and accept contributions to Meteor, like the brittleness of tests, and lack of contribution guidance. In fact these are problems internally as well, when people join the company to work on Meteor it is hard to learn all of these things.
Unfortunately, this isn’t a problem that can be solved quickly - it has to be a turnaround in the culture that got us to where we are now. In my opinion, this means:
Using more standardized testing tools to that we can take advantage of other stuff in the community like code coverage analysis, and to make it easier for contributors to write tests (who wants to figure out selftest and tinytest, and when to use which)?
Leaning more heavily on unit tests, which run much faster, over end-to-end tests which sometimes require uploading and downloading files to servers or building huge amounts of code
Splitting up the code and the release process so that, while there can still be Meteor version numbers like 1.2 and 1.3, the versions of major subcomponents (Blaze, Minimongo, Build Tool, Cordova, etc) are also meaningful and these parts have their own tests. It would be great if to submit a PR to the Cordova integration you didn’t have to run every single build tool test, and I think the only way to do that is to decouple those systems (which would also allow projects like meteor-electron to be more integrated).
I think part of the first step is to have new projects start out as independent parts. A small nod towards this is the Meteor Guide repo, which is partially an intent to move the docs out of meteor/meteor, where people have found it very hard to contribute to the docs. But we can do much better and we are working on it.
I think once we split up some of the components, it will be simpler to give people in the community commit access to take a more active role in development, since it will be more clear what the APIs are, what the goal of that component is, etc.
More than a year ago.