I’ve done some digging around and a lot of the threads regarding file system and how it works with Meteor seem to be pretty outdated, not to mention packages related to file storage/serving (i.e. CollectionFS). I was wondering if anyone here has deep experience with handling files in lieu of 1.4 or even 1.3 (I am currently on 1.4.1.1).
My questions are as follows:
Did Meteor 1.3/1.4 come with any changes regarding fs?
What is the general best approach to storing and serving static assets in light of Meteor 1.4?
I’ve seen many threads that say dynamically storing files to /public triggers a server upload, but I’ve tested this on local by manually copy/pasting a .png file into /public, and it only triggers a client refresh with the console message Client modified -- refreshing. Would this hold true for files added during runtime, and would it hold true in production?
Currently I am trying to stay clear from S3 or any other third party CDN’s to keep a low budget, and also trying to stay clear from storing files into Mongo.
It’s very simple: you should not store changing data in public or anywhere else in the meteor folder. Only thing you could use is the tmp folder when needed. But most of time you can replace that with streams so that’s also an exception.
Static assets like images used in your app, logo, icons etc off course should be stored in public. They don’t change while the app is running. To serve them the guide advised also the use of an external cdn: https://guide.meteor.com/deployment.html#cdn
S3 is the cheapest solution in most cases actually. For sure when you put cloud flare in front. Off course you could also build your custom data storage when needed but actually that would be quite some work and I am not sure it would be cheaper.
How much data do you expect?
Edit: also to understand it a bit more why: for example galaxy does not allow you to write to the meteor app folder. This makes the software much more secure. For example the issue with many Wordpress install being hacked and starting to host malicious files simple cannot happen. It will host only the files you delivered.
Next reason is: in general you run multiple servers, so how would you sync all those files? How do you get all those files when starting a new server? All those issues are prevented by just using something like s3.
No, the cheapest solution is to store it on the server you’re paying for anyway to run Meteor on. Also, local file storage really isn’t that complicated.
How much s3 can one hour of troubleshooting something like collectionFS buy? Also, if you want performance, you’re running on an ssd vm with (usually) limited storage. So, why not s3?
Serving static files is usually the lowest rung of the performance-needing activities a webserver is wont to do. If your server is taxed by doing that then you won’t have much joy with something much more strenuous like Meteor or MongoDB anyway.
You also have full control over the files locally - which means you can do Cron jobs for batch processing and stuff.
Also, you don’t have to deal with REST APIs and authorization issues.
If you run a little app on one server where you actually are allowed to write then yes that might be simpler at first. In reality that’s in platforms like Meteor and for sure Galaxy not the case.
If you host it by yourself on a server you can do it with Meteor, write files outside of Meteor and write a script which sends them to the client. If it works for you great. But it’s not the recommended way to do things.
The main difference is that you see a server as something which stays for longer time. In reality webservers get recycled, scaled up, scaled down etc. For example if you push an update to Galaxy it creates a new server, moves the users to the new server and when finished removed the old server.
That’s why, even on a single server on that kind of platforms, you just cannot store your data on the web server. Same happens on AWS and all other platforms which allow scaling. You also see this in for example the 12factor app: https://12factor.net/processes explained in more detail on the why of this.
You don’t need a script to serve up files. It’s about 2 configuration lines in nginx which then serves files from any directory you want. (You want nginx anyway for SSL)
I.e. Meteor lives in /var/www/meteor and the static files live in /var/www/static. With my config I can then serve up files from the static directory under http://myserver/static and everything else gets handled by meteor.
And if you need scaling then you could of course move to AWS. On the other hand, it’s not that hard to scale a simple fileserver as well.
If it works for you all great @rhywden! Off course if you setup a server for hosting the images outside of the Meteor directory that’s fine. Just answering the question of this topic and why it is that way.
What is the general best approach to storing and serving static assets in light of Meteor 1.4?
Something external, not Meteor itself.
And to be complete on the other questions:
1 Did Meteor 1.3/1.4 come with any changes regarding fs?
Nope, the build system is being worked on which is kind of related but there are no new features or developments to host dynamic files.
3 I’ve seen many threads that say dynamically storing files to /public triggers a server upload, but I’ve tested this on local by manually copy/pasting a .png file into /public, and it only triggers a client refresh with the console message Client modified – refreshing. Would this hold true for files added during runtime, and would it hold true in production?
As mentioned above: Don’t do it. Put it outside of Meteor.
There are many more related topics about this which you can find via the search.