Embarrassing Homelab Mishap
So I made a bit of a dunderheaded mistake with how I setup my backups of two servers.
I set them up to rsync folders to my homelab NAS over SSH. That’s great. I even filtered out a bunch of unwanted files using an exclusions file.
However, I didn’t bother to exclude the rysnc folder or the tar folder inside; where the data is staged and then compressed. By design it leaves a copy of the data on the local drive as a tarball.
It was supposed to keep 31 of those, and then start dumping the oldest. This was assumed to be okay–even on the small disks ~40GBs or less; due to the total size of backup being around 40MB.
Well when you fail to exclude the tar folder you end up with a compounding backup where a 40MB backup becomes an 11+GB backup in short order.
Needless to say, today two of my machines ran out of space and various things ground to halt. Whoops. Well I cleared off the cruft and fixed the backup script. I also decided to keep 7 tarballs not 31. I also setup a blowoff valve to empty the tar folder if the space gets to or above 80% full. The local copy is not that important–just convenient. All the data is ultimately stored on the NAS.
This was poor system design on my part and bit me right in the ass. Sharing as an object lesson in thinking through your designs and how simple mistakes can lead to lots of cleaning.
