Rolling your own Meteor js environment

If you’re reading this, you too likely had that “ah-hah!” moment when you stumbled onto Meteor. You quickly downloaded the install script and had Meteor running in a matter of moments. That install script took care of all the details, installed node.js, mongodb, and even launched your app on a local port for development. You were off and running.

It is a beautiful moment for all Meteorites.

The first day I started using Meteor, I had an idea that became our first production Meteor app (a tool for resource planning that we use in the course of our own day-to-day web consulting). We’ve done several since, and it’s been a rush for sure.

And there are certainly a few things we’ve learned in the process. In this blog post, I’d like to share with you the journey we undertook in finding a robust, scalable, affordable and multi-host environment to ensure that our app didn’t crumble under the load of daily traffic.

We failed miserably until we got it right. So we’d like to help prevent you from suffering the same fate.

As it stands, you have a few options when hosting a Meteor js app:

Use Meteor's hosted solution

You can use “meteor deploy myapp.meteor.com” in your project root directory to spin up your app in a development environment that’s maintained by the awesome folks at Meteor. It’s a simple process. The problem is that it is explicitly stated that this is not for Production apps and you don’t get multiple hosts. So pretty quickly, you’ll need to find a new solution once you go live.

Note: Meteor’s hosting platform Galaxy is coming to beta in the coming months. It’ll likely stay in beta for a while with a disclaimer that it’s not for Production apps. When it’s ready, it’s going to be glorious, but until then, you’ll have to find an alternative solution.

Use a node.js platform hosted solution

Use “demeteorizer” to bundle up your application as a standalone node app and deploy it to one of the several Node hosting platforms like Heroku, Nodejitsu, or Modulus. This will certainly work if you only need one server (dyno, servo, or whatever each person calls it). But once you need to scale beyond that, you’ll bump up against the fact that none of them offer sticky sessions in a way that your Meteor app needs.

Don’t let their marketing fool you. Some claim to offer sticky sessions. They are lying and when you run into problems and in the face of proof (an ip log showing the same visitor hitting multiple servers), they’ll shrug their shoulders and say, “we don’t know Meteor”.

If you do go this route, the main problems you’ll encounter are:

• Their cloud infrastructure is aging and ailing in nearly all cases. The machines are slow and have very little memory. You’ll notice that your app is slower than it should be, load on the servers is ridiculously high, and when you start getting a lot traffic to a single host, memory becomes an issue. You’re typically capped at 100-300 megs of memory for your app and if you exceed that, they’ll either restart you or let your server (and app) drown.
• We encountered issues with our app reloading randomly, over and over and over again from time to time when we had more than one server in our app. We tracked this down to the fact that each network request for assets was hitting different servers and thus getting multiple sessions on each app startup. This mostly happened when you hit the app for the first time. Sticky sessions?….yeah right.
• You have no idea what version of Node each server will be running upon deploy. And neither does the hosting company. That’s right. No idea. Just a bunch of machines floating out there with different versions. So you have to demeteorize your app with the lowest version of all of them. This can hamstring you.
• You could end up with zombie processes. With Nodejitsu, we once had random servers executing old code that were not receiving new deployments, but were still executing. It took weeks of back and forth with support to figure out what was going on and even then, they still couldn’t fix it. I just had to delete my app and redeploy in a new namespace…(what??).

  We were faced with the problem of having our app drown under the load in a single server, or inflicting random restarts on our users when we had more than one server. It really sucked.

• You are at the mercy of the hosting company’s upgrade schedule. If you keep track of it, you might be okay - but our app went down at least once due to upgrade issues on Nodejitsu’s infrastructure and when asked why we were notified beforehand, they said something to the effect of, “oh, it shouldn’t have had any problems so we didn’t announce it”. Ah….

Roll your own solution on the cloud platform of your choice

If none of this sounds attractive to you, then you do have another option. It’s the place we ended up at: we just rolled our own stack. And it was easier than you might think.

The basic line-item breakdown of our Meteor js stack is:

• Cloud hosted servers on Rackspace (you could easily use Amazon too).
• One load balancer directing load with nginx.
• Two hosts to which you deploy your app also running nginx *and* Node so as to maintain the SSL endpoint (very important - notes below).
• A standalone mongodb server firewalled to only accept connections from the Node hosts because mongo servers are a bit insecure to begin with.
• A bash deploy script that pulls the latest commit to the branch of your choice from a git repo, bundles it up with “meteor bundle” and pushes it to the hosts and restarts node.

Don’t worry - we’ve done the heavy lifting. But before we get into the details, let’s first examine some of the stuff that’s good to know, and which you no doubt glossed over in your zeal to start cranking out code (because naturally, Meteor makes it so easy to do this).

When you sit back and think about it, the magic of the Meteor command line tool does a few major things for you.

• It launches a mongo instance
• It launches a node.js instance
• It sets a few environment variables to connect the above services
• It relaunches node should your app change or crash

Let’s dig into each of these in a little more detail:

Your mongodb server

You’re going to need a mongodb server. The battle over hosted vs cloud vs roll your own is a debate for the forums and IRC. In the end, you need pick one. To start, using a hosted solution will probably save you some time and headache, so you might start with that (check out MongoHQ). On the flip side, for what it’s worth, we’ve noticed having a mongo server in our local cloud environment helps with performance (latency, machine power, etc).

If you want to roll your own on a cloud server like Rackspace or EC2, here’s a link to jump start your efforts:

http://docs.mongodb.org/manual/tutorial/install-mongodb-on-red-hat-centos-or-fedora-linux/

Security Alert: If you’re new to MongoDB and setting it up yourself, the main thing I’d tell you to look out for is permissions (it’s not like MySQL which is mostly locked down from the get go). Mongodb comes shipped rather open in terms of security. They have a concept called the “localhost exception” which should be addressed in the first few steps of installation. Lock down your firewall, create an admin user, and make sure that you disable the localhost exception. At least then you’ll be protected until you figure everything else out.

http://docs.mongodb.org/manual/tutorial/enable-authentication/
http://docs.mongodb.org/manual/tutorial/add-user-administrator/

You will also need to create a database and a user to access that new database (although technically you don’t need to create a database. It’ll be created for you when you use it).

Spinning up node.js

You will also need to install node.js. The version of Meteor you’re developing for/on plays an important role (though keep in mind that the danger of not being mindful of this might not rear it’s ugly face right away). From the start, I tried to pick the node.js version that is run by the Meteor command line tool when you are bundling via “meteor bundle” - mostly in order to save my sanity and eliminate the possibility that this is the culprit when diagnosing issues.

• Meteor < 0.6.5: node.js - v0.8.18
• Meteor >= 0.6.5: node.js - v0.8.24

Bundling your App

When you’re ready to push to production, you’ll want to use the Meteor bundle command. This will package up your app files for deployment. Once deployed on your server, you’ll find a main.js file which will be your entry point. “node main.js” will launch your app.
Environment Variables

Your meteor app will minimally require two environment variables:

MONGO_URL mongodb://USER:PASSWORD@HOST/DATABASE
ROOT_URL https://MYSITE.COM

You can set them locally in the current shell context using export:

export MONGO_URL=’mongodb://USER:PASSWORD@HOST/DATABASE’;
export ROOT_URL=’https://MYSITE.COM’;
node main.js

We preferred to set these variables up in a simple bash configuration file that is executed just before we launch the node application. Of course (as mentioned above) you could set them up in the bash_profile (or equivalent shell) as well, but I thought setting them just in time for running the application made more sense.

“Is there a chance the track could bend?” Restart your app when it crashes, my friend.

Lets keep this simple, use this - https://github.com/nodejitsu/forever.

Deploying your app

We actually concocted a bash script that deploys a branch from our repo to all active node.js servers in our stack. We’d encourage you to do the same as it saves time and makes deploys much easier. You can also do things like automatically ensuring that your staging environment always deploys your “staging” branch and your production environment always deploys your “production” branch.

Node Fibers

The trick that many Meteor developers don’t know is that Meteor doesn’t use node.js in a traditional manner. It uses a package called Fibers that forces node.js to function differently. More importantly, for deployment it’s important to keep in mind that Fibers is platform dependent. Ideally, you’ll want to run Meteor bundle from the same machine you intend to run node.js from. This will ensure the correct version of node-fibers is installed.

If you do build it on a Mac (or whatever else else), you’ll need to use NPM and update the node-fibers package within the bundle. When doing this, make sure to replace the node-fibers module on the target machine, otherwise you’ll simply install the wrong version each time.

I’m probably not explaining myself very well (I am not an eloquent man), so just take a look at this link and heed the advice of the highest voted answer: http://stackoverflow.com/questions/13327088/meteor-bundle-fails-because-fibers-node-is-missing.
When the traffic gets to be too much: Load Balancing

Here’s something interesting: On all the hosted platforms we tried, our app was drowning under the “load”. However, on a 1-gig Rackspace instance, load has rarely jumped above 0.2 and has never locked up due to memory. What a difference having a machine that’s not something out of 1997 makes!

So it could very well be that you don’t need more than one server if you roll your own solution. But if that’s not the case and you’re lucky enough to have the traffic to support multiple node.js servers, then picking the right load balancer is important.

Keep in mind, Rackspace actually has it’s own load balancing solution for it’s servers - which won’t work for Meteor apps (darn…). You’ll need support for Websockets, and you’ll also want to have Session Affinity, or Sticky Sessions in case an old browser is using SockJS.

Here are some options you could go with:

• Nginx
• Varnish
• HAProxy

We have fairly extensive experience with Varnish, but ultimately opted to go with Nginx because it was super simple to set up and configure - and is pretty powerful.

Oh...right. Security. (A bit on SSL termination)

Since most sites are incorporating SSL as a default now, using SSL with your Meteor app should be part of the puzzle. I spent a bit time trying to find out how to get node.js to handle SSL termination until I found it a lot easier to terminate the SSL connection using a proxy like Nginx (or you could use the above mentioned proxy servers). All you need to do is put a small single purpose web proxy in front of your node server (with its sole function to handle in the incoming SSL and send off an unencrypted connection to node.js). Make sure you do this on the same physical machine, not across networks - otherwise you could be exposing the unencrypted data to prying eyes. To be clear, I’m proposing an nginx server on both your load balancer *and* each node.js server you have for your app. The Nginx instances on each talk to each other and the only unencrypted transmission occurs from the node.js server’s Nginx instance *to* the local instance of node.js. Easy peasy.

Here is our nginx load balancer conf file:

upstream backend {

	ip_hash;
	server [NODE_X_IP]:443 
}

server {
      listen      80;
      server_name [DOMAIN];
      rewrite     ^ https://$server_name$request_uri? permanent;
}

server {
    listen		443;
    server_name		[DOMAIN];
    access_log		/var/log/nginx/access.log;
    error_log		/var/log/nginx/error.log;
        
    ssl on;
    ssl_certificate /etc/nginx/ssl/server.crt;
    ssl_certificate_key /etc/nginx/ssl/server.key;

    location / {
    	
        proxy_pass https://backend;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        # WebSocket support (nginx 1.4)
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

    }
}

And here is our ssl termination nginx conf file:

upstream backend {

	server 127.0.0.1:8080;
}

server {
    listen	4433;
    server_name  "";
    access_log	/var/log/nginx/meteorapp-access.log;
    error_log	/var/log/nginx/meteorapp-error.log;
        
    ssl on;
    ssl_certificate /etc/nginx/ssl/meteorapp-server.crt;
    ssl_certificate_key /etc/nginx/ssl/meteorapp-server.key;

    location / {
    	
        #access_log off;
        proxy_pass http://backend;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        # WebSocket support (nginx 1.4)
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

    }
}

In conclusion:

The ease of setting, developing and deploying a Meteor app can do us a slight disservice in that it prevents us from knowing how everything works. This is of course a blessing and a curse because it lets you focus on cranking out your app rather than these finer points. It lets you be creative rather than bogged down in technical details - and this is a beautiful thing. But ultimately, you have to pay the piper sooner or later. At some point, you have to know these things or you can get stuck. Understanding these backend systems will help you understand your app better. It’ll help you debug more efficiently. And ultimately, it’ll ensure that your app stays up rather than crumbling with success.