Project Details
The problem
An always-on home server running dozens of interdependent services (smart home control, security cameras, media, personal automation) has a specific failure mode that’s easy to under-build for: something crashes at 3am, and without real self-healing, it just stays down until a person happens to notice, which for a security camera or a wake-up automation is exactly the wrong time to be silently offline.
What was built
- HomeKit-integrated security cameras with local, on-device AI motion and face detection, processed on the box itself, not sent to a third-party cloud service to be analyzed.
- A large containerized service stack covering home automation, media, and personal-automation workloads, all on one box.
- Health-checked automatic recovery for every service: a crashed container or a hung process gets detected and restarted without a person intervening.
- A priority-tiered boot sequence. After a crash or reboot, services don’t all fight for resources at once. A staggered, priority-ordered restart brings the most important things (home control, security cameras) back first, deliberately ahead of lower-priority background workloads.
- A standing resource-contention rule: HomeKit and camera reliability always win over lower-priority jobs, enforced structurally rather than hoped for.
Technology choices, and why
Local AI processing for cameras, not a cloud service. Motion and face detection run on the box itself. That’s a real engineering cost (running detection models locally instead of paying for a hosted API) taken on specifically because security-camera footage is exactly the kind of data that shouldn’t need to leave the property to be useful.
Health-checked recovery over “restart on any exit.” A service that’s running but not actually working (hung, unresponsive, silently broken) is a different problem than one that’s crashed outright, and needs a different check to catch. The recovery layer checks real health, not just process existence.
Monitoring what a user actually experiences, from where they experience it, learned the hard way. A real incident exposed the gap in checking only “is the service marked active locally”: several real, live-facing services went down together when a lower-level networking layer failed, and every local health check still reported healthy, because from inside the box everything looked fine. The failure only showed up on the path a real outside user would actually take. That incident became a standing design rule: a health check has to exercise the same path a real user would, not just confirm a process is technically running, and any newly discovered blind spot gets real monitoring added as part of fixing it, not filed away as a follow-up.
Outcome
A genuinely self-healing system in real daily operation: automatic, health-checked recovery across the whole service stack, a crash-safe staggered restart sequence, and a standing reliability hierarchy that consistently protects security and safety-relevant automation ahead of everything else. The monitoring approach itself has real incident history behind its design decisions, not just theoretical best practice.