I am thinking about closing the CollectionFS project/github org, having a project this large demands a lot of effort. It has suffered during the development of Meteor.
CFS reached nearly 62k downloads and 50 cfs related packages.
Reading messages on the issue tracker I know there are people using it and helping each other in getting running with CFS, but I also see / hear the frustration regarding maintaining the project.
I strongly feel that the best maintainers of open source projects is the people that depend on it.
Eric (aldeed) and I have created cfs v2 - but lacked time and proper tools for writing tests in Meteor - no tests no v2 - We are both busy at Dispatch working on amazing things with an awesome team, but we don’t use CFS.
With the limited spare time and 30 packages (without counting cfs or ground db) to maintain, and an upcoming potentially package breaking meteor release (getting the first issues already…) - I need to cut more packages.
Thank you for all the work you’ve done for the community, Morten. I certainly understand your frustration, and can’t blame you. This is a big loss, though.
I totally understand you have other obligations in life, so thank you very much for your generous work, and for providing support for as long as you did.
Okay. So… because GridFS is necessary for blob storage of PDFs and DICOM images, the healthcare industry has had an eye on the CollectionFS project since Morten and I had our talks about file uploads 3 years ago. I know of at least a dozen companies that have CollectionFS integration somewhere on their roadmaps, and will be impacted by this.
Unfortunately, most of them are not active on the forums; but I’m going to start reaching out to them for corporate and financial support in managing this deprecation. I just posted a message to the Healthcare Working Group, to hopefully raise some awareness over there. Will also talk with the Radiology folks at OHIF, and figure out a plan if we want to fork it, volunteer to maintain the existing org, etc.
I don’t directly use CollectionFS, because I’m working on other parts of the EMR stack other than DICOM. So, I’m not the best person to maintain the package. But if we have to park it somewhere, Clinical Track could absorb it.
Also, just last month Clinical Track figured out it’s FDA strategy and got a basic suite of Validation/Verification tests across all 25+ of it’s packages and it’s 3 reference apps. It’s just a beginning, and there’s lots of refactoring and upgrades to do. But we have the release-track wide testing harness in place.
We can do the same for CollectionCF.
It will take time. Probably a few months before we can green light things for refactoring. But the Gagarin tests should be able to handle all of the integration testing; and the Nightwatch tests will be able to handle any complex UI testing scenarios. We’ve done file uploads with Nightwatch in a few different scenarios; and I have the recipes in the Cookbook or hidden away in various project repos.
But, if you need QA testing on CollectionCF to maintain it; and you need to offload work to other people… I’ll try to figure out a way to absorb the responsibility. It’s too critical of functionality for the healthcare space to loose.
Morten…
what are the core packages for minimal functionality?
are the packages at/near version 1.1.0.3?
will you continue as an org member if we can help set up tests?
how willing are you to add member/owners to the org?
or would you just rather just transfer ownership so it’s off your plate?
If the only use of CollectionFS is for GridFS, I migrated to vsivsi:file-collection. Quite easy to move over (reusing GridFS collections if you match the nam) and it has no problems with multiple meteor instances like CFS has.
Why deprecating/closing? Why do not you add some maintainers and you let the community do the work?
So what happen to the people that are currently using it?
Comparing to the number of other packages on atmosphere 62k users are on the high end.
So traction is not the problem. It sounds strange that the first alternative when the main contributors run out of stamina is to deprecate their work. If that was the standard Node.js would have been deprecated when Ryan Dahl decided that he did not want to keep working on it.
Code is usually deprecated because there is not enough traction, or there is a better alternative. These are not the case for CollectionFS, at least I am not aware of a good alternative that has comparable features.
A better route, when there is traction and no alternatives, is to find maintainers that will keep the work going.
And that would be my expectations here.
For the time being, I can help managing the issue and PR on the project: closing the ones that are not relevant and merge the pull requests that will fix some of the problems.
I know there were a lot of expectations from the community about CollectionFS.
I think many of us did test and tried to help improved the lib.
Frustrations are shared from both side I think.
Thank you for your great job for the community.
As @awatson1978 stated, GridFS is a huge requirement. But with Meteor v1.3 and direct npm integration, couldn’t we use existing npm lib (if any) for file upload to gridfs store ?
There are existing upload packages and fileservers on npm and in the 1.3 of Meteor it will be much easier to leverage these tools.
Ideally we should use aws api’s - If multiple versions of files / images is needed it should use SQS listing on S3 events and have that trigger a lambda function (small node functions with imagemagick available) to transform into a new format.
Ideally we would have up/downloads run using native on ios/android and webworkers in the browser.
That said, newcommers might find it difficult to setup things like that.
Ideally cfs would just be a node module with an observable interface?
8 Open governance
Problem is that we are all too busy working on things, cfs requires a ton of test writing - we haven’t been able to leverage existing test tools + writing tests for existing code requires a complete understanding of how cfs works - not a small task.
If anybody want to lead the project it’s ok with me - but think twice before deciding, it’s a potentially time consuming task.
Well, this package is of importance to me. There might be other packages on NPM, but none that integrates well with Meteor and Meteor 1.3 doesn’ t really change this (it was already quite easy to use NPM packages).
I’ve been diving in the code recently to add azure support ( https://github.com/CollectionFS/Meteor-CollectionFS/pull/905 ). Actually, while diving in the code, I already rewrote parts of the core to ES6. I don’t have the time to lead this project, but I’d be willing to be a co-maintainer if we can find a couple of other people willing to join in.
I have been trying to create my own version of CFS. I forker the repo and updated “Csf:” everywhere. I applied a pull requests. However when I hit node publish.js everything falls apart.
With the Clinical Meteor track, I’ve been focusing on identifying the commonly-used APIs and getting test coverage on them in preparation for bigger refactorings.
For instance… with iron:router, we’re only putting test coverage on Router.route() and Router.go(). We figure that a lot of the internals are going to change as we migrate to FlowRouter or whatever is the next shiny bit of tech. But if those two APIs are kept consistent, that will satisfy 80% of the use cases out there.
Personally, that’s what I want to do with CollectionFS. Identify a few of the core APIs, get them under QA coverage, then figure out where we need to go from there. We might swap out the REST API? Fine, lets not put all that under QA; but lets identify the inputs/outputs so we know what to implement with the next REST package. Etc. etc.
@raix thanks for all the work with the CFS packages and the community! It’s a shame, but 1.3 does provide good timing and an appropriate motivator. We (ongoworks) would be happy to partner up on maintaining the CFS repo/org if others find it worthwhile. In the meantime we’ll probably just fork as well.
