Panic moment with Home Assistant

Today I want to tell you about the panic moment I’ve had these past few days with Home Assistant. So there I was, chilling in Valencia during this last long weekend, when I noticed that the devices that show up in my Home app coming from Home Assistant weren’t responding. I opened Home Assistant and everything seemed fine, but there was an alert about the last backup, and as you know I use the Google Drive Backup add-on for Home Assistant, which is an absolute gem… but the thing is, this add-on had an error because the last backup had failed… and it wasn’t a connection error with Google or running out of space on Drive… it was an I/O error, an Input/Output error… and that’s genuinely scary, it usually means there’s a problem with the disk or the controller or something like that… anyway, because I’m an idiot, I thought a full restart might fix the problem… and nope, nothing of the sort, the NUC stopped responding and never rebooted again.

The High Availability Problem

As you know, I’ve always said that what I don’t like about Home Assistant is its lack of a high-availability setup. In the end everything depends on whatever system you have Home Assistant installed on, which could be an old PC, one of those NAS units that some of you have at home because you’re a bunch of pirates… I don’t know anyone who has a NAS and only uses it to back up their photos and videos, but anyway… the point is that Home Assistant can also be installed on a Raspberry Pi or a NUC… but yeah, only on one of them. You can’t have a Home Assistant install on a NUC and on a Pi, or on the NAS and on another device, so that if one fails, restarts, or whatever, your whole system keeps running. There are some approaches out there from people who put the Home Assistant database on a SQL database and store that database on some NFS system so it’s accessible from two installs at once… sounds great to me, but at the end of the day the Zigbee dongle or the SkyConnect is only ever plugged into one of the two. Sure, if one fails you just have to move the Zigbee dongle to the other one and the devices will detect that the physical address of the dongle – which is basically a unique MAC address type thing – is the same, so they’ll connect without knowing which host is behind it, where that dongle is actually plugged in… but that requires manual work. With HomeKit, however, it’s not like that. If you have devices connected directly to HomeKit via Wi-Fi or Thread, even if a HomePod goes offline, as long as you have another one it’s not a problem. If your Apple TV breaks but you also have a HomePod at home, the system will ‘balance’ the device hub from one to the other completely transparently to the user. Obviously if your router dies you’re fucked, but even in that case you can have several mesh routers, like my Eero setup, and one of them would also take over control of all the Wi-Fi devices if the other one broke. Obviously if you have a bridge it’s not like that – if you have a Hue bridge and it breaks… well, the bulbs lose their connection. Also, if you buy another one and have 2, you can’t pair all the bulbs with both so that if one breaks, the other takes over control… each one has its own associated bulbs and they don’t share them with each other. Well, this is basically what happened to me. When the NUC stopped responding, all the devices that reach HomeKit – the ones that show up in the Home app but come from Home Assistant – stopped responding. 35, no more, no less. The Matter devices didn’t notice a thing and could still be managed directly from the Home app, same as the native HomeKit ones, but all the Zigbee stuff went straight to hell.

Not dead, just out partying

So the thing is, when I got back home after a few days of silent panic from Valencia (kids, lesson learned: don’t restart Home Assistant if you’re not nearby to hit the button or make sure everything boots up properly), I hooked the NUC up to a monitor and saw, to my dismay, that it wouldn’t boot – it just showed a message saying no bootable device was found.  The first thing I thought was that the disk had died, even though it’s a good SSD. I checked that it was properly connected and then booted the system with a bootable Linux USB stick I keep around for… well… for my own stuff, and from there I could see the disk perfectly fine, it recognized all the partitions, the checks said everything was fine… but it wouldn’t boot, it seemed something had happened to the boot partition, and the system recognized the partitions but didn’t detect any of them as bootable. So I downloaded the Home Assistant OS image and installed it from scratch. Then I booted up a completely fresh, brand-new Home Assistant install, downloaded the Google Drive Backup add-on and restored the backup… and after rebooting, everything was already working again without any issues. The process itself took me less than an hour, once the problem was identified it was all pretty quick, but the uncertainty of not knowing why it happened is something that really bugs me. Who’s to say it won’t happen again tomorrow… or next week… and there’s nothing you can do about it. Sure, I could have another NUC ready to go so I’d just need to move the Zigbee dongle and restore the backup like I mentioned before, but that’s nowhere near an automatic solution. So this has reinforced my resolve to keep hugging HomeKit close, gradually swapping devices out for Matter ones and flashing the Shellys with HAA so I can keep removing devices from Home Assistant, or at least have HomeKit be the one publishing them to Home Assistant, so I can have things on 2 different platforms and run tests, rather than the other way around. It’s true that fixing it only took me 1 hour… but the cold sweats and the fear I’ve had over the last few hours every time my wife walked near a light switch, I wouldn’t wish on my worst enemy.