When trying out new things, especially things that you usually need to do in a live production system, it can be helpful to first have a low-stakes disposable environment to practice in. It’s also helpful to be able to practice at a much slower pace, and with time to think and digest, at your own speed.
Just like in Kung Fu, practitioners start out extremely slowly, with a wooden dummy. Then with practice they get faster and faster until the reflexes and pattern recognition are sufficiently developed for full-speed motion.
This is why I wanted to create the mongo-dojo, mostly for myself. Databases are often very important and critical components of running production systems, and we do our best to avoid touching them. This is unfortunate, since focused time on task is where we actually get better at things.
So instead of just reading about how to do things in a mongodb replicaset (which is good, though insufficient), we’re going to actually DO things.
The Scenario
You’re running a mongodb replicaset on VMs (for “reasons”), and it turns out you need to create a rolling index. This is usually when a new field gets introduced to the data, or maybe a specific query has been introduced that is now causing performance problems. For whatever reason, you need to build an index on the two secondaries, then step down the primary and build an index on that.
The only problem is you’ve never built an index before, and you’ve never worked with mongodb replicasets before, and you’ve never stepped down a primary before, and you’ve never touched a production database before, and so on and so forth.
This procedure involves numerous different steps (as detailed in the repo), which need to be performed in sequence with close monitoring of the system.
But never fear! We’re going to practice exactly that!
And not only are we going to practice it…if anything goes wrong, we can just destroy the whole thing and start again from scratch. The worst that can happen is you need to spend some time restarting the whole thing.
MASSIVE CAVEAT: If you happen to be running this on the same computer you use for work, absolutely make sure that you haven’t done something silly like alias the mongo shell command to automatically connect to a production database…in which case this dojo might very well turn into an actual fight!
The Exercises
As discussed above, the repo walks you through how to create a rolling index.
You also get practice upgrading the version (like the above exercise, this uses VMs, since it requires a restart of the mongod process).
You can compare the results of a query run on an unindexed field with the same query after having created an index (this one’s not a rolling index, so you can just use the container set up).
And lastly, you can see what actually happens when a long-running query spikes the heartbeat time when it’s run on an unindexed field…then you can fix it by adding in an index.
All of this is integrated with prometheus and grafana, so you have a realistic monitoring set up to actually see how the stats change with these different actions.
A well-used dojo
The current iteration of the mongo-dojo isn’t exhaustive in terms of the different tasks you’re asked to do, but it served me well in getting to feel comfortable with a mongodb replicaset.
It’s got two modes:
- VMs
- containers
basically because the VMs are closer to what I deal with in real life, and two of the exercises required process restarts anyway, and I didn’t feel like messing around with process managers in docker. For the other simpler things, containers are more than adequate.
I also like this project as a proof of concept for how we can build training tools for ourselves to practice learning objectives in a very specific and measurable way. There’s a difference between knowing what to do, and knowing how to do…and this is a great example of being able to hone skills in the latter.
micro materials blog 