Things you need to know when scaling Meteor JS
It’s a jungle out there, it is. And chances are that (in your manic hacking frenzy) you’ve committed some transgressions that will come to bite you when you finally float your app onto the stormy seas of Production.
How did we chance upon this knowledge? Well, first and foremost, through blood, sweat and tears from our own production resource planning app crashing repeatedly during its first week. Through this chaos we formed some assumptions about what was going on and remedied the situation with some hot-fixes.
But for confirmation, we tracked down Meteor co-founder Nick Martin at a Meteor Devshop (if you're in the Bay Area, you should be going to Devshops!) and managed to get a few questions answered.
In other words, we went straight to the source.
Are you wasting precious pipe maintaining a channel of data unnecessarily?
So before we get started, let’s make a distinction between anonymous, unauthenticated users and logged in users. Their experiences are most likely completely different - and (importantly) there are some considerations that go along with this.
For the purposes of this blog, we’re generally referring to anonymous / unauthenticated users. You’re likely going to have way more of them than logged in users (this includes normal users and search engine bots) and they generally just see either static content - or at the very least some limited set of data that a normal logged-in user would see.
With that out of the way, the question becomes, how do we separate these two types of visitors in terms of the data we serve them? What magic does Meteor.com use to serve millions of visitors to its site (with impressive simultaneous numbers during its launch) on the same servers that it has everyone else deploying test apps to via ‘meteor deploy’?
If you follow most of the examples on the web, you’ll do something like the following when defining a collection and permission query (taken direction from our own code, so you’ll have to use your imagination for your own application):
if ( (Meteor.isClient)){
Deps.autorun(function () {
Meteor.subscribe(
"boardCards",
Session.get("current_board"),
Session.get('viewAssignmentsGuestKey')
);
});
}
Cards = new Meteor.Collection('scheduler_cards');
What are doing here? Well, we are telling this code to run only client side - and secondarily, we are asking the client to open up a channel of communication to the “boardCards” subscription and listen for any updates from the server in case it has any new stuff for us. We’re also defining our “Cards” collection.
In the case of our app, you only have boardCards if you are logged in and have made a few. If you are an unauthenticated visitor though, it’s impossible to have any yet. So you wouldn’t need the subscription and you wouldn’t need the collection.
But the code above wouldn’t know that. It would define the collection client-side and faithfully listen for updates that will never come, consuming memory all the while. Multiply this by ten or more collections and you’re adding a healthy bit of memory (protip: Meteor by default currently opens a new session for each tab a user has open) you’re giving away unnecessarily from a server that doesn’t have that much to begin with (modulus.io for example will restart your app once you’ve consumed over 400 mb. Heroku will restart it after something like 200).
To compound matters, Meteor will maintain an open connection for nearly 15 minutes after the last time it hears anything from the client just in case the visitor is going through a tunnel on their mobile phone (or something like that) and is just temporarily ducking out of the “conversation”. So your memory for each connection will stay allocated until the user doesn’t come back for over 15 minutes.
This can get expensive.
So what’s the solution? It’s simple. Don’t give people stuff they don’t need (What are you, made of memory?). How do you do that? Wrap your subscription and collection definition code in a few more conditions.
Namely:
if ( (Meteor.isClient && (Meteor.userId() != undefined || Meteor.Router.page() != 'front_page'))) {
Deps.autorun(function () {
Meteor.subscribe(
"boardCards",
Session.get("current_board"),
Session.get('viewAssignmentsGuestKey')
);
});
}
if ( (Meteor.isClient && (Meteor.userId() != undefined || Meteor.Router.page() != 'front_page')) || Meteor.isServer) {
Cards = new Meteor.Collection('scheduler_cards');
}
So what’s the above code doing? Well it’s the same as before, but we’re checking a few things.
In a nutshell:
- Are we running as the client and are we authenticated? Do we have a userId? If so, then subscribe to the boardCards subscription.
- What’s this other stuff? Meteor.Router.page()? Well, we have some instances where we need to show an unauthenticated user some subscription based data. But this is never on the front page. We use Percolate Studio’s Router package to handle our routing so we know if we’re on the front page or not, so just don’t do all this jazz if you’re on the front page (which handles the bulk of the traffic to our site).
In the second condition block, we’re defining the collection only if we’re either server-side, or if we’re authenticated client-side.
Does that make sense? Basically just expose your data where it needs to be exposed.
Incidentally, Zoltan at Percolate Studio recently submitted a pull request to add the capability to detach an active session from particular tabs on a client. So for instance you could make it so inactive tabs don’t listen to the server until they are active again - thus saving memory overhead server-side. This pull request will likely be merged into core very soon, so you’ll have this baked right in to Meteor before long.
This extends to code too
So something else that’s interesting: Why expose code to unauthenticated users too? You can add the same checks to blocks of code that need not run for anonymous users. Be careful with this though. At least right now, we’ve found that some code that we do this with doesn’t end up getting run properly once the user logs in - so make sure you have a proper testing plan in place.
Selective querying/updating helps
When we asked Nick for some tips that might help resolve common mistakes that meteorites make, he pointed out that it’s important to add an explicit '_id' value to your queries and updates. It should be noted that this isn't an index in the sense of mongodb - but rather, it's baked right into core. The meteor query processor is able to take a shortcut when both queries and the updates have '_id' explicit. This will speed things up tremendously because that’s the main thing core can recognize immediately in the queries and results it returns. So whenever possible, add an {“_id”: “xxxxxx”} to your queries and updates as a condition in your selector. This isn't to be confused with the performance enhancement of using '_id' as an index *in* the mongodb sense of the word. This will imbue it's own (less dramatic) performance improvement. So just keep all this in your bag of tricks.
And don’t make static stuff reactive, while you’re at it
So some pages / content don’t need to be reactive. We’ve discussed making queries static in a previous post. But you can do this for templates too using #constant regions. Check out the Meteor docs here.
In conclusion:
If you’re looking for general performance tips, check out our previous blog "Improving the Performance of your Meteor JS projects".
Kittens shouldn't have to die
Our goal with this post though was to illustrate the impact of some things that you could easily overlook. Keep in mind that every time you waste 10 megs of unnecessary server memory, a kitten dies. We should strive to prevent this. To that end, we hope this post has been helpful!
