What better way is there to kick off a blog than tell a good story. As very few people might remember, I originally started posting here in 2020 and had built up a decent track record over the months. That was until March 2021, when it all came crashing down…
OVH datacenter outage
On March 10, 2021 I woke up to a myriad of notifications informing me about my sites being down. I promptly confirmed this and while I’ve already had experienced reliability issues with OVH in the past, the alarming difference was I had trouble finding any information on the cause of the outage or any sort of maintenance scheduled for the day.
When the true cause and extend of damage was eventually revealed… let’s just say that’s not what you want your data center to look like.
As I was only using a single VPS instance to host the blog, the damage has been done, the site was gone. And unfortunately, so was the data.
Now, a blog is pretty simple beast and I did have copies of all the articles stored locally – only to realize they’ve gotten lost when I was switching machines earlier that year. Embarrassing really.
So here’s what I took away from it:
Don’t get yourself caught up in marketing terms
I’ve never really thought about it much beforehand but the truth is, VPS and Cloud services are usually advertised with very high availability as one of the key features.
The truth is, the code still needs to run somewhere. And if you’ve only got one instance, when that machine goes down, so does your infrastructure. So it’s really up to you to ensure the safety of your project – be it through purchasing additional services from your provider or committing to maintaining your own solution.
Whatever you do, don’t ever use the same data center your primary infrastructure runs on for backup purposes. A mistake too many people seem to have made in this case, and I feel very sorry for them since they actually paid for a service that has proven utterly useless in the end. Don’t make that mistake.
Being intentional about data redundancy
Admittedly, it was not a mission critical project – but being the one affected, loosing data hurts just as much. So first and foremost, don’t treat ensuring data redundancy as an after thought.
It might sound obvious but I promise you, I wasn’t the only one to loose data in that outage. Some of my friends and co-workers and even some notable companies were in the same boat.
Choosing the right solution for the job
There’s more backup solutions out there than I can name, so it’s important to recognize your requirements in order to make the optimal choice. In my case, the thought process was:
- I use Ansible to maintain my infrastructure as code. Hence I don’t care for backing up the entire server, as I can set up an identical environment from scratch in a matter of minutes.
- At the moment, given my (un)popularity, availability is not a huge concern to me either. Loosing the site for a few hours or even days is far from ideal, but tolerable. Servers cost money.
- I do still need to back up the actual site content.
After the outage, I’ve seen a bunch of articles pop up on how to set up redundant Kubernetes clusters with fancy failover mechanisms, letting your site run in independent data centers all over the world. While the geek in me smiles anytime I read things like that, I can also recognize it’s overkill for my needs and only really solves the reliability/availability issue – you should still have proper backup.
Instead I played it simple and went with Jetpack Backup. As the site runs on WordPress, the plugin does exactly what I need – backing up site content – and it does it daily, in a fully automated process. Restoring the backup is also fast and easy.
Setting it up took a bit of work but that’s only really due to how I like to run things on my server, which I might cover in another post. And it comes with the additional perk of being free as long as I remain an employee at Automattic.
As I’m sure everyone of us has dealt with some issue like that in our careers, I’d be thrilled to hear about it too if you’re willing to share!