My Quest To Production; eventual [success] :-)

deployment
1

Hi everybody,

as I stated recently, what Meteor is missing for me is a solid deployment system. My words [could not have been more correct] / [where a self-fulfilling prophecy], as I was forced this very week to go to https production. Let me relay my quest into Mordor for you… :slight_smile:

1) DDP problems @ the client
First of all, I “deployed” a test-version to a server I have in my office. It seemed to work fine - until I got the clients office, where it failed miserably… :cry: Nothing worked, not even logging in. Some digging in the debug-console revealed the problem: websockets where being blocked. Boohoo. This turns out to be a known problem. Apparently, this can be remedied by using httpS, as the SSL-encryption prevents the proxy from seeing that a package is actually a websocket-call. So, I needed to set up a server with HTTPS.

2) The Trying with different HOW-TO’s
I started with this, after googling: “ubuntu meteor deploy”. This used mupx. Googling mupx showed me multiple versions on github, may depricated etc etc. Not confidence inspiring, really, so I googled on. DO is highly regarded, and this how-to looked fine to me. So I started with step one,and carried on. I installed a VPS with Ubuntu 16.04 (always use the last LTS-version, right?) and carried on with apt-gets and what not, until the very last step. That failed.
Turns out, using the last version was not correct in this instance, as 16.04 uses systemd, which does not support start but wants something else. Rather than trying to figure out how to convert the script to systemd-rightness, I whiped the server, installed 14.04 and tried again.
After spending a blissfull afternoon trying to work out how to a) run on a separate user, but b) installing node [using nvm, helped there by folkes on #IRC] but then of course being the wrong user, c) not having the wrong rights to run the script, and d) almost throwing my DELL through the window, I was getting more interested in MUPX by the minute.

3) Bliss with MUPX (that line will rhyme, but only if you read it out loud…)
Lets try with mupx, shall we? :slight_smile: MUP(X) is installed via npm. That was not obvious to me, but I got there eventually. (I’m still not quite clear why, but ok.) There are multiple versions of mup, mupx etc, but I ran first mup, which failed, then I found this how-to. I started to follow that, after wipeing my VPS and reinstalling 14.04.

4) Installing the server
Installing the server seemed easy enough: mupx setup. But you run that from your local machine and not from the server. Took me another try to get this right. (Yes, I know, it does not state that you should run it from the server, but it seemed logical to me!)

5) mupx init
Takes a very long time (+30 seconds anyway) without any feedback to my by now very tired eyes. But then: green lines! Yes! Then, mupx deploy. RED lines! NOOOOO!
A few gotchas in this as well:

  1. you need a fully qualified domainname in the json;
  2. you need to be able to login with root-access.

Root access? I have been told you are a very bad dog if you bite in root…

6) Root this out!
MUPX really want to log n as root. Ubuntu of course, works with sudo; mupx will not. Stackoverflow to the rescue; how hard can it be? Logging in as root: done! I can log in as root!
Of course, I could not ssh in as root. That is something else… Took me some time again to realize this. Fixed.

7) SUCCESS, well, sort of!
Now, if i command mupx deploy, I get… success, success, success RED FAILED; the last step: verifying deployment, fails. With the last line: To see more logs type 'mup logs --tail=50’
If you type this, it is a bit of a bummer, because it will give a new error. (This happens only once, because next time the app run, it will echo the log from your app - but not the first time!)
Ok. Well, it seemed to work, so lets try the browser. And it works!
Of course, we do not have a db, and SSL is not working yet, but at least I have a working app…

8) Uploading the db
I come from a LAMP-platform, so dumping a db and loading it is easy as pi. Turned out, the pie was hidden in in a container, so it could either be eaten allready, or be still be there - no way to tell…
Ok, so I need to use meteordump locally - can do that. Except that is not installed. But Meteor comes with mongo pre-baked? Apparently, without meterdump. You need to install that, not by npm, but with apt. Not from the standard repositories, but using this methode. Done.
Creating a dump now will dump to a directory, and dump every collection in a new file. No worries, seems to be standard operation.
Upload to the server using scp, got it covered…
Now, reading the dump back into your database on the server, poses the problem with the container. Google was getting tired of me by now, but I was allowed one more query for the day. And I got an answer.
And yes, I could login now, and had my application working!
One full day, until this blissfull moment.
But SSL still remained…

9) SSoooLong! (And thanks for all the fish…)
According to the mupx-how-to above, installing a SSL certificate was just adding a few lines to the json, putting the certificate files on the right place, and off you go. Small problem: I got 3 files from my registar, and mupx expected two. It wanted a certificate and a key, and I had cabundle.crt, certificate.crt and certificate.key. Well, I did not ask for a bundle, so I took the certificate.crt , the certificate.key file, and put them in place. SSL ERROR on deployment…
Luckily, it was a bright new morning, so my new google allotment was active… you need to combine the two files into one!
With sweaty fingers… mupx deploy:
x Verifying deployment: FAILED

But it works! It even works with SSL!

I am still not sure where the error comes from. But my SSL is working, and my websockets are DDP-ing like never before. I’m still 2 months from deployment, :yum: so I booked a 6 week holliday! :sunglasses:
Of course, the last is not true. :cry:

Oh, deployment is very easy. It only took 1 month from my life in energy, and nearly 2 days in realtime.

But I got there :slight_smile:

This document might help someone else. I cannot by any stretch of the imagination be called an expert (as is now clear for experts, who probably stoped reading as paragraph 2). But it just might save anyone some time.

More importantly: I wanted to show how someone not an expert can strugle with this. I know I did!

Warm greetings,

Paul

2 Likes
2

@pwiegers Sorry to hear you had so much trouble. I still don’t understand why you don’t want to use a service like Galaxy? This whole thing would have taken you maybe 30 minutes, and that’s including creating an account and a database with a separate service.

3

, said the car driver to the mountaineer! :sunglasses:
Two serious answers:

  1. I must be absolutely certain (regulations) that I keep the data at least in Europe, but rather the Netherlands. Sorry, no shortcuts allowed :frowning: I read somewhere that now you can select AWS/europe since recently, but I read so many bad things about getting Meteor on AWS that I tried the other route first. And more people experice than would care to remember, once you choose something, changing to another is hard.
  2. I really “do not get” the “buy mongo somewhere else”-thing. If I buy Meteor hosting, I want Mongo to be included, thank you very much. As I said in the other piece I link to above, part of the lure of Meteor for my was the lack of complexity with the db. This does not help :frowning:

The sad thing is: I would really love to support MDG! :heart_eyes: But I really nead those two points remedied. If I am confident enough about 1) then my next version will run on your servers :sunglasses:

Again, I love Meteor, so I will really try to support your organisation too!

Paul

2 Likes
4

To be fair to Meteor you’d have just as many issues trying to deploy an app on a bare server in most languages / frameworks. There’s a reason there’s loads of one-click cloud hosting solutions that do all the hard work for you in exchange for lots of money.

However, it’s great that you want to and have done this yourself, and I think it can actually be quite fun… if you like that kinda thing. I’m just going to stick some general tips below that hopefully you and/or others will find helpful. If you don’t want to spend some time getting set up initially you should really just cough up the $25 / month for Galaxy and $18 / month for Compose.

  1. Yes this is an annoying problem with websockets and proxies, but you need https anyway for any real application. You can’t transmit login details, user information etc in plaintext, that’s madness, and even for a static site with no personal information it leaks every page a user has browsed to anyone who’s motivated enough. Luckily LetsEncrypt has come to free our wallets from the slavery of the big certificate authorities and offers certificates absolutely free of charge. There’s a relatively simple CLI tool for this (CertBot) and once you’ve automated the process (which you need to do because the certs only last 90 days before you need to renew, in order to strongly encourage you to automate it…). You are still free to cough up cold hard cash if you want a long lasting certificate from an old-school CA (or if you need EV certs).

  2. Ubuntu 16.04 has slightly messed up all the existing tutorials out there but hopefully they’ll get updated soon, it’s not actually a massive change but say you have a script in /etc/init/ or /etc/init.d/ I think you’d run something like systemctl enable my-script.sh and that’ll make it work with systemd.

With this stuff it’s vital that you record everything you’ve done and make it repeatable otherwise you get a month or two down the line and you need to spin up another server and you’ve completely forgotten how to do it. I would recommend learning how to use one of SaltStack / Ansible / Chef / Puppet for configuration management. You spend some time writing config files that describe the software and configuration you need on the server and the configuration management tool will apply it to your server(s) perfectly time and time again. No need to be typing commands from an online guide over SSH and hoping you’ve done it right. What I tend to do for a new setup is run through once manually over SSH, recording what I’ve done as a SaltStack state, then I boot up a completely clean VPS and test the state until it’s working perfectly. Now you don’t waste time trying to keep all that in your head and all your servers are identical so you don’t get weird issues on just one server.

To address the last part of 2), NVM was the right thing to do here. We use NVM to install Node and PM2 to manage running applications, and both can be installed locally on a non-root, non-sudo user to keep them from being a security risk.

3-7) I’ve never used mupx, so can’t really comment on it. I imagine root is required to install mongo and such, and also to run the app on a privileged port (i.e. below 1024). The setup we use is haproxy running as root and nginx, node etc. running as the non-root user. I trust (relatively) that haproxy is secure, but I trust less that my own code is secure.

It may be helpful to read the mupx source so you can work out what’s happening, or even modify it to suit your needs better.

I set up my own deployment system and command-line tool so I can just type a command with the name of the app and a git commit sha / branch and a target server or servers and it will build the app for linux locally, upload the tarball to S3 and then call a salt command to initiate a script on the app servers which downloads the specific tarball and installs it before reloading the app in PM2. You could even write a script that just uses SCP/SSH to copy the files and unpack them.

  1. Database wise if you’re deploying a production app I wouldn’t even bother trying to roll your own mongo replicaset unless you’re sure you have the time or manpower to manage it. Use https://www.compose.io and everything including backups will be taken care of. You can even manually (or via their API) download backups to your local machine as an extra protection. I don’t think the new MongoDB+ works with Meteor though, you’d have to use the classic MongoDB for now, which is cheaper anyway.

Final thoughts, even when you’ve set up servers the work doesn’t end. You need to keep on top of security patches and deploying the latest versions of your apps and making sure you don’t run out of space (or inodes) on disk (delete old builds/deploys hanging around on the server, meteor generates lots of small files which in my experience use up all your inodes before your disk is even 60% full).

Hope this essay helps, and let me know if you want me to expand on anything!

2 Likes
5

Aside from latency (which nobody seems to run into), I don’t see the big deal about having to host the DB somewhere else. I like having a company who does nothing but DBs handle the DB.

I suppose it would be nice if galaxy could automate/integrate compose into their service, but it may not be worth it really. If I remember correctly setting up is 1. signing up to compose and creating a DB (5 minutes) and 2. adding the DB URL to your app.

1 Like
6

ARGH - 2 days in and just got to your step #6.

I’ve had mup working before and it worked very well… trying to get the latest mup with docker is killing me.

1 Like
7

To your point 1, just recently got this e-mail from the Galaxy team:

Looking to use Galaxy for Meteor hosting but need more performance or privacy? Want to run Galaxy in an AWS region beyond US-East or EU-West?

Galaxy is now available in a dedicated cluster configuration for enterprise users! Enjoy the “zero devops” convenience of Galaxy in the region of your choice without compromising security or scalability.

A Galaxy Dedicated Cluster delivers:

  • Complete infrastructure isolation for your Meteor apps - no sharing with other Galaxy users
  • A reliable, enterprise-grade runtime with predictable scaling characteristics
  • A low-latency environment ideal for high-traffic apps
  • The ability to run in any AWS region of your choice
  • Enhanced support including 24x7x365 1-hour guaranteed response time for critical cases

Reply to this email or contact galaxysales@meteor.com to learn more about how to deploy to your own “private” Galaxy dedicated cluster!

Not using Galaxy? Get started with the worlds most popular Meteor hosting solution.

-The Galaxy Team