Why You Want to use WiredTiger with Meteor 1.4

When you want to rapidly prototype ideas that then turn into large applications, Meteor and MongoDB are a hard combination to beat. Recent versions of these tools – Meteor 1.3 and MongoDB 2.6 were no exception.

As your application grows, it will eventually need to handle an increasing amount of transactions, generating more and more data each day. With more frequent writes to the database, you could encounter performance issues.

In its latest iteration, Meteor 1.4 has become more useful and efficient than ever. One of Meteor 1.4’s most attractive features is support for MongoDB 3.2 — and its default storage engine, called WiredTiger. This new storage engine is designed to help you meet the demanding needs of Meteor applications by squeezing more power from your existing hardware.
In fact, it would be easy to argue that they work together better than ever before. So, I thought I’d share some context for WiredTiger along with simple instructions on how to set it up.

Project Ricochet is a full-service digital agency specializing in Meteor, Apollo, and GraphQL.

Is there something we can help you or your team out with?

Why WiredTiger?

One way to handle a higher volume of data transactions is to distribute the tasks among more servers. That concept – called sharding – can be accomplished easily with MongoDB, because it was designed for scalability. Another way to handle more data would be to explore ways to process it more efficiently. But in this case, is it possible to get more out of each machine?

The MongoDB team set out to answer that very question. The result of that effort turned out to be WiredTiger, a NoSQL startup they acquired at the end of 2014. This new storage engine was designed to maximize the efficiency of multi-core CPUs and the large amount of RAM available in most modern hardware.

How WiredTiger Helps You

1. Higher Throughput with Document-Level Concurrency Control

   Consider your classic traffic bottleneck. Whether the cause is a construction project or accident in the road, multiple lanes must merge into one. Well, a very similar kind of congestion would occur when using MongoDB’s original storage engine, MMAPv1. Write operations to the same collection can only happen one at a time. Other concurrent writes would have to wait their turn. WiredTiger improves this by offering document-level concurrency control. This effectively adds more 'lanes' to the collection. Your application can now perform concurrent write operations as long as they're updating different documents. Servers that have multiple CPU cores gain even more advantage here.

2. Lower Storage Costs with Compression Support

   WiredTiger can compress your data and help reduce the amount you pay for storage. Yes, with compression there's a performance hit. However, you'll enjoy higher storage I/O scalability as fewer bits are read from the disk.

   There are two open source compression algorithms to choose from. Snappy has a lower compression rate with a minimal performance hit. zlib provides stronger compression but higher performance impact. You can choose from one of these methods or no compression at all. There's even a way to mix the two. You can have frequently accessed data assigned to storage devices with Snappy compression enabled. The less frequently accessed data can be stored on other devices using zlib compression.

You can enjoy these benefits with virtually no changes to your code. With MongoDB's pluggable storage engine architecture, you can read and write to MongoDB, no matter what storage engine is being used, with the same Meteor Mongo APIs. No coding changes are necessary when moving from MMAPv1 to WiredTiger.

Run with the Tiger

Since WiredTiger has been dutifully proving its strength out in the wild, the MongoDB team went “all in,” making it the default storage engine for 3.2. That means you can start taking advantage of WiredTiger in your own Meteor development environment right now. Specifically, Meteor 1.4.1 uses MongoDB 3.2.6 as the local development database.

New Applications

If you create a new Meteor 1.4+ application, WiredTiger will be set as the default storage engine except if you’re on Windows. There, MMAPv1 is still the default because the version of MongoDB that comes bundled with Meteor doesn't run WiredTiger on Windows. However, it is possible to run MongoDB 3.2 separately to use WiredTiger in this case.

Can Meteor scale?

Learn more about Meteor performance with our white paper on the "Top 10 Meteor Performance Problems" and how to address them.

Download White Paper

Existing Applications

For an existing application upgrading to Meteor 1.4, the local database should remain untouched. It will continue to use the original MMAPv1 storage engine. In this case also, you can make the switch to WiredTiger manually.

If you don't mind losing your existing development data, simply type:

> meteor reset

… and your Meteor 1.4 app will create a new, WiredTiger-powered database the next time you start it up. With your app running, you can confirm it's on WiredTiger with the mongo shell.

> meteor mongo
> db.serverStatus().storageEngine

{
    "name" : "wiredTiger",
    "supportsCommittedReads" : true,
    "persistent" : true
}

If you'd rather not lose the data in your local database, you'll need mongodump to export it. It comes bundled with MongoDB which you can download if you don't already have it. Once installed, with your app running, export its data with:

>  mongodump --host localhost --port 3001--db meteor

Next? Stop, reset, and restart your app with a new database using WiredTiger.

> meteor reset
> meteor

Finally, with your app running, import your previously exported data back in:

>  mongorestore --host localhost --port 3001 --drop

You should now be on MongoDB 3.2 using WiredTiger, with your data intact.

Using an External Database

If you're not using the bundled MongoDB that comes with Meteor, you'll need to specify which storage engine to use when creating the database. To use WiredTiger, type:

>  mongod --dbpath /data/db --storageEngine wiredTiger

You can confirm the storage engine by querying the startup log:

> mongo 127.0.0.1:27017
> use local
> db.startup_log.find().pretty()
{
    …
    "startTime": ISODate("2016-08-27T01:44:51Z"),
    "startTimeLocal": "Fri Aug 26 18:44:51.842",
    "cmdLine": {
        "storage": {
   	     	"dbPath": "/data/db",
   		"engine": "wiredTiger"
   	 }
    }
…
}

That's it. With this new setup, you should be able to achieve “cat-like” speed and efficiency.

Run Wild with WiredTiger

The MongoDB team suggests that WiredTiger can deliver up to 7x-10x performance improvements and that compression can help reduce storage needs by up to 80%. It all depends on your data volume, load, and storage hardware. Give WiredTiger a try to see how it can benefit your application.

Curious about how much it might cost to get help from an agency that specializes in Meteor & Apollo?

We're happy to provide a free estimate!

A bit about Marty:

Marty has a B.S. in Computer Science and over 15 years of software architecture, development, and management experience - building everything from large scale, enterprise back end systems to innovative client side apps using Meteor.js. In his free time, he loves playing hockey and imbibing Tim Horton's coffee when he's not playing with his three kids.