After working on Meteor-Files for 5+ years I realized it is difficult to use and you need to “understand it”. I’ll keep original Meteor-Files package up to date, but it will enter LTS stage after upcoming v2.0 release.
I’d like to ask for your input — what do you need as developer the most when uploading files within web application? Please, let me know.
Here’s my ideas for meteor-files-light:
Stick to the HTTP transport
Keep “progress” features on the Client
Keep “continuity” feature to be able to resume upload when connection got interrupted or on poor connection, like 3G
Keep onAfterUpload hook on the Server, to manage file after it’s fully uploaded
Keep onDownload hook on the Server, to manage access to the file
Keep onBeforeUpload hook on the Server and Client, to manage/abort upload;
Keep .remove, .load, and .addFile methods as part of Server API
Result of upload would be “downloadable” path to the file, and object with its stats like: name, size, mime-type
Get rid of Collection and its features — you’re in full control on where and what data to store
I have been using Meteor-Files with GridFS in a meteor app and it does the job quite well! I think the current API is a bit puzzling at first because it does not follow Meteor’s API. Here are some examples where the API is not great (I hope the experience I have to share is not the result of some laziness on my side):
It returns a FilesCollection object that has an insert and remove method, but the real Meteor collection object is FilesCollection.collection. The methods do not behave at all like Meteor collection ones, for instance, you should not wrap FilesCollection.remove in a Meteor method (took me two years to solve it, and by accident. For some reason, I was stubbornly convinced that I should use a method. Probably should be called upload/trash instead of insert/remove to make it clear that this has nothing to do with Meteor.Collection methods. Really anything but insert/remove. The next logical step would be to rename FilesCollection to FilesLibrary.
Related to the previous point: the fact that permissions are handled differently than the Meteor way is puzzling. Not clear how to enforce strict server validation, took me a long time to finally use onBeforeRemove (OK maybe I am dumb). Maybe there should be dedicated methods validateUpload/validateRemove.
It adds a lot of metadata by default (userId, isPDF for instance) which may not be suitable for all application (also maybe userId should be uploaderId?). Also file type detection is not reliable anyway: isPDF is true if extension is .pdf on upload even if the file is actually HTML. isPDF does not change automatically when you later rename the file to correct the file extension. I think this should not be part of the API anyway, having a working file upload/remove is enough, the metadata can be added by the user as needed.
@harry97 thank you for jumping in, good question, and good reference.
No, and yes
Yes, this already similar to CollectionFS and Reaction-File-Collections as it was inspired and followed idea of CollectionFS. I’m going to release ostrio:files@2.0.0 fixing open issues and matching Meteor@2.0 requirements, but keeping current API.
No, new “light” version package goal is to remove everything not related to actually uploading the file to a Server. After file reach its destination (a Server) you will be in full control of storing its reference in DB, moving it around or to 3rd party storage, detecting its type and metadata, renaming, etc. So, it would focus only on uploading the file, not managing Files.
@aureooms agree at every point. Since it has long history and was inspired dead by now CollectionFS it took many wrong turns. Today I want to freeze its API and enter LTS, after fixing few long-running issues and following meteor@2.0.0 recommendations.
For the new “light” version of the package my goal is to throw away everything not related to actually uploading a file. Like you mentioned unnecessary meta-data, hooks, and bulky/confusing APIs. Upload result would be “downloadable” url, on Server only path to actual file’s location, and its name. The rest would be under your control.
Hello,
We are users of your packages for many years now. This is a great package.
We extensively used the meta properties in the file collection to store all information we needed to “manage” the files. We could without problem use another collection (not included in your light package) to manage this (for this of course we need the hooks).
I agree with @aureooms that ext, extensionWithDot, isJSON, isAudio, … are not to be included in a light version of the package.
Uploaded files are named on the server : RandomId.pdf, will you continue with this approach or will you give the real name of the file on the server (how to manage two files with the same name…) ?
An important thing for us is to check rights when serving the file for download or display in the browser. Would you do this in your package or are you focusing only on upload ?
Another important thing : Both packages (Meteor-Files and Meteor-Files-Light) should be able to work on the same app. If so we could progressively implement the new light.
I decided to implement “light” version as two NPM packages (so it would serve needs of entire JS ecosystem, not only Meteor).
Client library would have everything related to upload
Server library would be implemented as middleware, responsible for receiving
Why two libs? Strip-off unnecessary code from the Client bundle, and be able to upload files to other domains dealing with CORS.
Download middleware will exist only in examples and tutorials section, as its implementation is relatively easy and individual to every app, as you mentioned — in some of the apps there’s a need to check permissions, other apps would need to stream video files, other won’t need download feature at all.
I’ll include examples for permission validation and many other cases. In the end — documentation and examples make open source packages strong, right?
But since you’re using features of Meteor-Files you should be good, there’s no need to switch to other package.
Hmmm, but if it’d focused on uploading only. Do you think it’s worthy of a package? Isn’t that something the developer should manage also? I don’t have a definite answer.
But If one were to manage the files on his own how hard could it be to manage the uploading part? Funny thing is I’m working on something similar recently where we trim Meteor-Files to the absolute needed.
If I can make a wish, I’d like to see a package that:
can config an s3 bucket almost automatically (cors, policies, other weird AWS magic - like mup does for nginx)
can sign upload requests so clients can upload directly to s3 without the server
can manage the state and metadata of the uploaded files in a meteor collection
can create permanent and temporary direct links to the uploaded files (e.g. check the user’s role in a method, ask the package for a file, send the user a temporary link created by the package)
@csiszi we have different goals over here. I’m pretty sure there are packages focused on direct upload to S3. My goals with 3rd party storage are same — keep it “storage agnostic” with examples and tutorials, but file would reach the server first. Removal won’t become a part of “light” version, and would remain under your full control, again, with examples and tutorials.
Surprisingly, not a lot of real-world apps need to download uploaded files, so upload middleware will live in tutorials section only as well.
Thank you for valuable feedback, it’s good to know that so many apps rely on AWS.
@harry97 there are not many packages doing that even outside of meteor ecosystem, that’s why I’m going to distribute it over NPM. Most of libs trying to solve every case shipping with support of 3rd party storage, resizing, processing, etc. taking precise control away from developers.
Btw if I’m wrong I’d more than happy to check out other “lightweight” file-upload libraries. Send me links
I can’t help you much since I’ve not been involved outside of Meteor stuff but multer is the only thing I can think of. I believe you’ve managed to sum up what your library would be focused in this comment which is a reasonable goal so I guess what remains is to double check any solutions out there trying to do the same even slightly because you can either choose to base your work off of theirs or even become a contributor and improve it.
On a side note, if you were to start with this project I don’t believe you should kick start it with a focus on the Meteor community since Webapp would make things a little difficult for you, just a speculation.
A slingshot option should be very helpful. If the target is being “lightweight,” it is good to have an option to remove the load on the server (meteor) and have the client directly upload to a 3rd party server
@dr.dimitru this is great news! One question: Will it stick with cookies for authentication or do you think there will be other options to utilize the tokens from localStorage in order to authenticate requests? I know for Cordova there might be not much of alternatives but maybe for Meteor web apps and pwa?
I’m not dropping ostrio:files though, since it manage such detail as authentication internally, at the same time I remember how this was complicated to fit all edge cases in ostrio:files package.
As I’ve mentioned before “download” middleware won’t be a part of “light” version of package, and I’ll cover it with examples, where we can authenticate users before downloading a file via query-string, Authentication header, etc.
For authenticated uploads you will be able to pass custom header(s) (as it was with x-mtok) or query-string and build your own authentication in onBeforeUpload hook.
