@aureooms agree at every point. Since it has long history and was inspired dead by now CollectionFS it took many wrong turns. Today I want to freeze its API and enter LTS, after fixing few long-running issues and following meteor@2.0.0 recommendations.
For the new “light” version of the package my goal is to throw away everything not related to actually uploading a file. Like you mentioned unnecessary meta-data, hooks, and bulky/confusing APIs. Upload result would be “downloadable” url, on Server only path to actual file’s location, and its name. The rest would be under your control.
Hello,
We are users of your packages for many years now. This is a great package.
We extensively used the meta properties in the file collection to store all information we needed to “manage” the files. We could without problem use another collection (not included in your light package) to manage this (for this of course we need the hooks).
I agree with @aureooms that ext, extensionWithDot, isJSON, isAudio, … are not to be included in a light version of the package.
Uploaded files are named on the server : RandomId.pdf, will you continue with this approach or will you give the real name of the file on the server (how to manage two files with the same name…) ?
An important thing for us is to check rights when serving the file for download or display in the browser. Would you do this in your package or are you focusing only on upload ?
Another important thing : Both packages (Meteor-Files and Meteor-Files-Light) should be able to work on the same app. If so we could progressively implement the new light.
I decided to implement “light” version as two NPM packages (so it would serve needs of entire JS ecosystem, not only Meteor).
Client library would have everything related to upload
Server library would be implemented as middleware, responsible for receiving
Why two libs? Strip-off unnecessary code from the Client bundle, and be able to upload files to other domains dealing with CORS.
Download middleware will exist only in examples and tutorials section, as its implementation is relatively easy and individual to every app, as you mentioned — in some of the apps there’s a need to check permissions, other apps would need to stream video files, other won’t need download feature at all.
I’ll include examples for permission validation and many other cases. In the end — documentation and examples make open source packages strong, right?
But since you’re using features of Meteor-Files you should be good, there’s no need to switch to other package.
Hmmm, but if it’d focused on uploading only. Do you think it’s worthy of a package? Isn’t that something the developer should manage also? I don’t have a definite answer.
But If one were to manage the files on his own how hard could it be to manage the uploading part? Funny thing is I’m working on something similar recently where we trim Meteor-Files to the absolute needed.
If I can make a wish, I’d like to see a package that:
can config an s3 bucket almost automatically (cors, policies, other weird AWS magic - like mup does for nginx)
can sign upload requests so clients can upload directly to s3 without the server
can manage the state and metadata of the uploaded files in a meteor collection
can create permanent and temporary direct links to the uploaded files (e.g. check the user’s role in a method, ask the package for a file, send the user a temporary link created by the package)
@csiszi we have different goals over here. I’m pretty sure there are packages focused on direct upload to S3. My goals with 3rd party storage are same — keep it “storage agnostic” with examples and tutorials, but file would reach the server first. Removal won’t become a part of “light” version, and would remain under your full control, again, with examples and tutorials.
Surprisingly, not a lot of real-world apps need to download uploaded files, so upload middleware will live in tutorials section only as well.
Thank you for valuable feedback, it’s good to know that so many apps rely on AWS.
@harry97 there are not many packages doing that even outside of meteor ecosystem, that’s why I’m going to distribute it over NPM. Most of libs trying to solve every case shipping with support of 3rd party storage, resizing, processing, etc. taking precise control away from developers.
Btw if I’m wrong I’d more than happy to check out other “lightweight” file-upload libraries. Send me links
I can’t help you much since I’ve not been involved outside of Meteor stuff but multer is the only thing I can think of. I believe you’ve managed to sum up what your library would be focused in this comment which is a reasonable goal so I guess what remains is to double check any solutions out there trying to do the same even slightly because you can either choose to base your work off of theirs or even become a contributor and improve it.
On a side note, if you were to start with this project I don’t believe you should kick start it with a focus on the Meteor community since Webapp would make things a little difficult for you, just a speculation.
A slingshot option should be very helpful. If the target is being “lightweight,” it is good to have an option to remove the load on the server (meteor) and have the client directly upload to a 3rd party server
@dr.dimitru this is great news! One question: Will it stick with cookies for authentication or do you think there will be other options to utilize the tokens from localStorage in order to authenticate requests? I know for Cordova there might be not much of alternatives but maybe for Meteor web apps and pwa?
I’m not dropping ostrio:files though, since it manage such detail as authentication internally, at the same time I remember how this was complicated to fit all edge cases in ostrio:files package.
As I’ve mentioned before “download” middleware won’t be a part of “light” version of package, and I’ll cover it with examples, where we can authenticate users before downloading a file via query-string, Authentication header, etc.
For authenticated uploads you will be able to pass custom header(s) (as it was with x-mtok) or query-string and build your own authentication in onBeforeUpload hook.
@dr.dimitru Exactly! The API should have two methods upload and trash. It should give two hooks to validate usage of these two methods.
For simplicity I think you should continue using a MongoDB collection to publish what files are available and this collection could be freely updated by the user, to add metadata for instance (I don’t know what would be a reliable way of preventing the user from calling insert or remove on this collection, or breaking the database by updating library specific fields with update). If you do not do this, you still need to have a way to list what files are available for garbage collection purposes (for instance uploads that succeeded but that were never linked to anything in the user’s system).
Furthermore I suggest that supported 3rd party storage should have implementation (plugins?) rather than documentation only (if someone is willing to do maintenance). In an application I work on I use file storage for instances deployed on my own hardware because I don’t want to upload sensitive data to the cloud and I want to simplify backup/restore/migration procedures by storing everything in the database. Therefore I use GridFS and I find that having to write your own hooks for such a standard solution feels a bit shaky. I would be more confident using a plugin that occasionally gets updates from the community that are then automatically deployed on the next dependency upgrade. Low-level hooks (afterUpload, afterTrash) should remain to allow for unsupported storage solutions.
@aureooms well… what you’ve described sounds ideal, and I wish to do that.
Ideally that should be bare-bone “upload” library with bunch of plugins for each and every storage solution, as well as framework (Meteor in our case).
At the same time that would require a lot of maintenance from my end, which, well… equals time. At this moment I contribute and maintain open source packages used in commercial projects I care about. Open source itself gives me joy (like chatting with you guys here ), but it’s far away from full-time work, atm I contribute as much as I can, and will in the future.
I’ll build “light” version with plugins in mind, — with simple, but rich API + docs and tutorials.
I would really appreciate the package, and I like the idea of a small, stable interface that abstracts away that single, tricky problem.
The extensive documentation and examples you’re envisioning are icing on the cake – which is, of course, the best part of the cake. It makes it so that the package comes in two parts: the core part that gets imported, and the super flexible part that gets cut-paste-modify’d from the documentation according to the use case.
I was going to suggest Meteor-Upload for the name, but perhaps something less Meteor specific if you’re going to publish on npm. Still, I think shifting the focus from Files to Upload might be an idea.
Also: I’m just integrating with Meteor-Files in a project for the first time at the moment, and I’m grateful for the package. the documentation, and the examples.
@colinf I’m glad you are sharing the vision of lightweight bare-bone upload library.
And I’m glad you’ve found a use in Meteor-Files library
Do not hesitate to ask questions over Meteor-Files package via GitHub issue or forum mentioning me.
