Why shouldn't I upload files in the public directory?

help
1

Hi,

I’m using Meteor-Files to let the client upload images into the /public directory of my Meteor App. I noticed that when I do it, my page reloads because of hot code push. After searching for a solution, I found people saying that it is a bad practice to let users save data on the “public” directory..

I would like to know why? I didn’t really found something about it. I don’t want to store my images in a 3rd-party storage service right now. I want to keep files on my file system.

Another question: when I upload the images in another folder on my file-system, I can’t load the images because of an access denied exception. It seems images has to be in my app folder, but then I have that hot-code push reload issue when a user upload an image. What do you suggest? I am on Windows.

Thanks!

1 Like
2

If you upload a file to /public everyone can access these files. If you want to only allow signed in users to access their own files, you have to choose another solution.

3

You said you do not want to use third party solutions and I understand you completely. You could use OpenStack Swift, which is an object storage just like S3 but that you can install yourself. I suggest you use the docker image curtis/swift-onlyone as it will take care of everything for you.

Using something else that your meteor application to manage/store/serve your user files have many many benefits.

  • If your app crash, the file server is still available (current uploads will continue).
  • If your file server crash, the application can still run and be used.
  • Meteor is not good at serving files, so serving user’s files will impact your application performance, and the file service itself.
  • Storing files in Meteor’s public folder make them available to the entire world, you might not want that.
  • File handling and Server applications do not require the same server specifications, so you’ll want to separate them later.

EDIT:

I realize that, while giving better insights on your other options I did not answer your question at all.

Why shouldn’t you store user’s files in public?
As it was pointed out, any file in public will be available to the world.
The public folder is bundled with your app, it is considered to be part of your source. It is meant to store resources to be used by your app, such as your logo, custom icons and fonts etc…
Just like your database, the user’s files are dynamic data that should be separated from your code, they are not required by your application to run. They are used to store information collected by your application and very much act in pair, but their content is normally not following your application in the different environment like development, staging and production.
When you will rebundle your app after some changes, you’ll have a new public folder content your images and font files, but not all the files uploaded by your users while you were working, so you will have to make sure that you are not deleting those files.
Even if your uploaded files are to be seen by everyone, you’ll certainly use a database to keep track of the uploaded files, their upload date, owner, size, tags, etc… That means that even if you keep those files around in dev/staging/production you will also need the full database… Merging the database changes when pushing to production will be even harder than just checking in on a folder…

2 Likes
4

Thank you ! That’s the answer I was looking for. I’ll have a look at OpenStack Swift.

5

Your welcome.

I’m using swifft (https://www.npmjs.com/package/swifft) as JS client to interact with the storage from meteor. You could use it only on meteor and use methods to proxy the file upload/download but you’ll certainly want to dig into using the formpost module and tempurl they’ll allow to give explicit tokens to a client so they can directly access the storage server for a specific time… Just like you could do with S3

6

In addition to what others mentioned, I think there’re security, performance and scalability concerns when sharing user uploaded files on your main app server.

  • Security because those files will share the same app server thus increasing the surface area of potential attacks

  • Scalability because if users keep uploading large file they might exhaust your memory

  • Performance because the upload process will share the same CPU bandwidth as your app server

1 Like
7

What do you guys think about using GridFS ?

8

I’m using GridFS, with CollectionFS. I think storing the files in the same way as the rest of your data is pretty nice for portability.

1 Like
9

I had considered GridFS, it actually can be very neat to use for SOME use cases… It really depends on what kind of file handling you’ll be doing. If it is just to store avatars, sure go with it.

In my case, I needed a way to reliably receive, store and use big files. GridFS wouldn’t have cut it as my meteor server would be put to a crawl while receiving an upload. The files then need some post-upload processing, and are done using a command line tool that can only receive a filepath, so I’d need to download the file from Mongo and store it on the filesystem to then process it and reupload it to mongo.

Now instead of having a simple one-shot backup solution for my database, I’d need one for user data and another for the files as I can’t handle them exactly the same way if I want to be efficient. I might also prefer to backup the small data way more often than the files, as they do not change as often.

Another point that put me away from GridFS is the cost of Mongo storage space VS object storage. Mongo size is more pricey per MB than object storage, so it’s another way to optimize my expenses.

—With all that said, even if it seems I’m very much against GridFS, I want to point out that I’m using it for a personal project that manages pictures. Displaying 1000 picture thumbnails, according to some filters, from the meteor server on localhost takes about 15 seconds, that includes everything from meteor subscription to the categories, pictures and receive the thumbnails. Note that it is running from a mechanic HD… If I switch to storing the pictures on disk and serve them using NGINX (no check for rights, if you got the fileID, you get the picture), it takes about 5 seconds but I loose the possibility to filter according to user rights. If I use a meteor method to get the files so I can filter the rights, I get real close to 15, about 13 seconds. When using a local Swift instance I get very close to 5 seconds, maybe 6 or 7.
For this project, the performance gain is not worth the loss of simplicity so I stuck with GridFS, and do not regret it.

10

not to mentinon, if you deploy your app to another service you’d have to migrate files. there is no probably guarantee that those files would survive a new code deployment.

11

Also, everything in the public folder gets loaded into RAM :stuck_out_tongue: (at least in dev, haven’t tested in production)

12

This behavior won’t happen on production servers.

Have a look at Cloudinary.com they’re AMAZING for media storage and offer a free tier that doesn’t suck.

Just FYI Meteor ‘sucks’ at serving content, CDN’s are highly recommended. That’s what I once read anyway. Cloudinary is this, plus so much more.

1 Like
13

Hello @dmonni ,

First, you have to answer the question - “Will be uploaded files publicly accessible to any visitor or access is somehow regulated and restricted?”. I’ll give an answer for both solutions, so you can pick one which meets your needs.

About app rebuild

By default (as mentioned in FAQ) files is placed to .meteor/local/build/programs/server and there is a reason behind it. Placing files into build/ won’t trigger app rebuild and hot code push.

Also in FAQ, we mention “Note: All files will be removed as soon as your application rebuilds” as if you stop/start app or change Server code it will trigger rebuild process and wipe everything under build/ directory. That’s why we recommend keeping files somewhere above meteor project directory, like /data/uploads/. Mind the leading / - it’s a root of File System.

Storage path permissions

You’ve mentioned permission issues placing files into very root directory, well it’s easy to solve with two Linux commands (will work for Mac too):

mkdir -p /data/uploads
chmod 755 /data
chmod -R 777 /data/uploads

Storage Options

If you don’t want to store files at 3rd party server and at the same time placing files to FileSystem is too boring for you, you’re free to use GridFS it let you store files at your own mongo, so everything under control. But really - storing files on FS is okay, and won’t eat server’ RAM at all.

Publicly available files

It’s a good idea to serve publicly available files with proxy/web server like Nginx or Apache as node.js ain’t a perfect solution to serve files. As you don’t need to build authentication and permissions logic - no need to involve node.js here.

There is public flag available on FilesCollection#Constructor, which will tell to the library to build absolute URLs to the files, so files can be served with Nginx, Apache or any other proxy/web server, just point its root to the upload directory or vice versa.

Files with access control

We have multiple ways to control access to the files: protected, interceptDownload, downloadCallback you can read about each of them and other security tips at our wiki.

I hope this information will help you to pick right solution and to solve your needs.
Do not hesitate to put your questions to our issues on GitHub, we have a great and experienced community there, someone will certainly help you to find a proper solution.

2 Likes
14

What? Proof please. On the prod or dev?

15

I discovered this when a guy on the forums was having trouble with Meteor crashing from having too many files in his public folder, so I tried putting a few thousand files in mine. It didn’t crash for me (seems like he had an OS issue, I think he was on Windows), but the startup took forever, and I saw that the RAM usage increased drastically.

As I said, I haven’t tried this in production.

Here, I did it again now, this time with larger files, which made Meteor run out of memory :smiley:

1009×350 97.3 KB

1 Like
16

everything in the public folder gets loaded into RAM

This statement is very confusing. At RAM goes a build cache, or more precise - hash table, and to get a file hash - file needs to be loaded to memory.

Anyways we should always differentiate public directory in a meteor project (dev) and public on a server (prod) where actually no public directory in a bundle, and public or root belongs to web/proxy server.