
The Alarm That Couldn't Ring
Building a heartbeat monitor for a narrow gap turned up a much bigger one — the alerting stack itself had been silently unable to notify anyone for over two weeks.

Building a heartbeat monitor for a narrow gap turned up a much bigger one — the alerting stack itself had been silently unable to notify anyone for over two weeks.

Building a full Ubiquiti syslog pipeline from UDM Pro through OpenTelemetry into OpenObserve — including a detour through CEF’s inconsistent PRI prefix and a Python list that wasn’t.

Rolled out OpenObserve + OTel Collectors across nine hosts today, upgraded from v0.14.7 to v0.70.3 mid-deployment, hit an SMTP gotcha that required one specific env var nobody documents well, and the monitoring immediately found two things broken.