# TVU host monitor

An **external** heartbeat for `product-demo.tvustream.com` — the self-hosted box that
runs the team's Gitea (git + CI + **npm registry**) and serves the design-system docs
site. Runs on the owner's laptop via `launchd`, every 15 minutes.

> **Lives here since 2026-07-30.** It started as a standalone local-only repo
> (`~/Documents/AICoding/VS_Code/tvu-host-monitor`, 3 commits, **no remote — one disk
> away from losing the knowledge below**). Folded into this repo because: two remotes
> for free (Gitea + GitHub, both already part of every commit here), every clone carries
> it, the docs that reference it (`docs/DEPLOY.md`, `docs/STATUS.md`) now sit beside the
> source, and no new external repo needed authorizing. The old checkout is inert — its
> plist path was repointed here — and can be deleted whenever.
>
> ⚠️ **Read [Coverage limits](#coverage-limits) before trusting this to page you.** The
> laptop it runs on sleeps; a monitor that sleeps is not an on-call system.

## Why this exists

The host already notifies on deploy problems: `deploy-hook/server.py` posts to Slack
(`chat.postMessage`, `SLACK_BOT_TOKEN` + `SLACK_CHANNEL`) when a repo goes from
healthy to failing, de-duplicates while it stays failing, and posts a ✅ on recovery.
Its alerting condition is `git pull`/`clone` success — a failed *build* is logged but
deliberately not alerted.

That covers "a deploy ran and went wrong". It structurally cannot cover two others:

1. **The host itself going down.** The notifier is a container *on that host*. It dies
   with the thing it watches. A watchdog on the same machine cannot report that
   machine being unreachable.
2. **The chain quietly stopping.** The notifier is event-driven — it fires when a
   deploy runs. No deploy, no message. That is exactly how the docs site sat frozen
   at `0.10.1` for two weeks (2026-07-22; see `docs/DEPLOY.md` in the design-system
   repo). Silence was indistinguishable from health.

Both blind spots got more expensive on **2026-07-29**, when INFRA-F73 moved the npm
registry onto the same host. Before that, an outage meant stale docs. Now it also
means every consumer's `pnpm install` fails. So the watcher had to move off the host.

There was also no monitoring stack on the host to lean on — probed 2026-07-29: no
uptime-kuma, prometheus, grafana, zabbix, nagios or netdata container running.

## What it checks

| Check | Probe | Healthy | Covers |
|---|---|---|---|
| `gitea` | `GET https://…/gitea/api/v1/version` | `200` or **`403`** | Git hosting, CI, registry host |
| `registry` | `GET https://…/gitea/api/packages/ux-team/npm/@ux-team%2ftvu-design-system` | `200` or **`401`** | Consumers being able to install |
| `docs` | `GET …/playground-dist/index.html` | `200` | The docs site being served |
| `freshness` | `Last-Modified` of the above | newer than `STALE_DAYS` (5) | A **stalled deploy chain** — the silent failure above |
| `gitea-ui` | fetch `https://…/gitea/`, pull an asset URL **out of the returned HTML**, fetch that | `200` | Pages that load but whose sub-resources 404 — the **blank-page** failure mode |
| `origin-ui` | same, against `http://…:3001/` | `200` | The other entrance, which since INFRA-F89 goes through our own `gitea-proxy` |
| `origin` | `GET http://…:3001/api/v1/version` | — **diagnostic only, never alerts** | Separating "reverse proxy / TLS down" from "Gitea down" once an alert fires |

**Why the two `*-ui` checks exist** (added 2026-07-31, INFRA-F89): on 2026-07-30 a
`ROOT_URL` change made Gitea render every asset under a `/gitea` prefix that does not
exist on the `:3001` origin. Those pages were blank for a full day and **this monitor
stayed green throughout** — every other check requests an API path, and none of them
ever fetches a sub-resource. The front page even kept answering `200` with the correct
`<title>`, so probing `/` would not have caught it either. Verified by replaying that
exact shape (a server answering `200` for `/` and `404` for the assets it references):
`gitea`, `registry` and `docs` all reported **up**, while `gitea-ui`/`origin-ui`
reported **down** — the old blind spot reproduced, and closed.

Two design points worth keeping: the asset URL is **discovered from the served HTML**
rather than hardcoded, so it does not rot when Gitea renames its bundles (and a probe
that 404s for the wrong reason is worse than no probe); and it prefers a real `.js`/`.css`
bundle over the first asset it sees, because a favicon would still catch a broken path
prefix but would sail past a missing script bundle.

Unlike the `origin` API probe, **`origin-ui` does alert.** The 2026-07-30 failure was
precisely an *asymmetry* — the domain entrance was healthy while `:3001` was blank — so
watching only the domain would have missed it. That port is also where our own `git
remote` and the deploy-hook still point. If `:3001` is ever deliberately retired or
firewalled off, demote this check to diagnostic rather than leaving it to cry wolf.

⚠️ Both entrances now traverse the same `gitea-proxy` container, making it a shared
single point: when it dies, `gitea-ui` and `origin-ui` go down **together** while Gitea
itself is still alive on `127.0.0.1:3003`. That asymmetry is how you tell "the proxy
died" from "Gitea died".

**Probe the ingress consumers use, not the origin port** (changed 2026-07-30): INFRA-F74
put Caddy on 443 in front of Gitea at `/gitea/` and made that the `ROOT_URL`, so it is
what every consumer `.npmrc` and every `git push` resolves. The proxy/TLS layer can
fail on its own — a monitor still probing `:3001` would sit green while every
`pnpm install` in the company fails. The origin port stays as a *diagnostic* line
(logged, never alerting, never state-tracked) because it may legitimately be firewalled
off later, and alerting on a hardening step would cry wolf.

**`404` is no longer healthy for `registry`.** It was tolerated back when the package
had not been published yet. It is published since 2026-07-30 (`1.1.1`), and an
injection that day proved the tolerance had turned into a false-green: a broken reverse
proxy answers `404` for every path, and the registry check stayed `up` through it.

**Why 403 and 401 count as healthy**: this Gitea sets `REQUIRE_SIGNIN_VIEW`, so an
anonymous `api/v1` request answers `403` and the npm registry answers `401`. Both are
proof the service is *up and responding*. Only a connection failure (`000`) means
down. This is what keeps the monitor **credential-free** — no token to store, rotate,
or leak, which matters because this registry is plain HTTP (see the TLS item below).

## How alerting behaves

- **Edge-triggered with de-dup**, mirroring the host hook: a banner on
  healthy→failing, a banner on failing→healthy, silence while a state persists. A
  15-minute interval therefore does not mean repeated notifications.
- **Four paths**, because a macOS banner only reaches you if you are at this Mac
  (⚠️ but see [Coverage limits](#coverage-limits) — "overnight" is exactly the window
  this monitor does **not** cover, because the laptop is asleep in it):
  1. **Slack** (`chat.postMessage`) — *optional, needs credentials*, see below. This
     is the one that reaches a phone.
  2. notification centre banner (with sound),
  3. `~/.claude/host-monitor/ALERT.txt` — exists **only** while something is down,
     auto-deleted on full recovery. Grep-able by a future session or another script.
  4. `~/.claude/host-monitor/monitor.log` — append-only history of every check.

### Slack — **configured since 2026-07-30**

`~/.claude/host-monitor/.env` (mode `600`) holds the `SLACK_CHANNEL`; `SLACK_BOT_TOKEN`
comes from the `~/.claude/daily-reports/.env` fallback, so one Slack app serves both this
and claude-work-report. Each variable is resolved independently, so the monitor's own
file only needs to carry what differs. **The concrete channel id and bot live only in
that env file, deliberately not in this repo** — they are internal identifiers, and this
repo is visible to the whole team on Gitea.

Two consequences of the channel chosen on 2026-07-30, owner-decided, recorded rather
than hidden:

- it is a **DM** (`is_im: true`), not a channel — so **only the owner sees an alert**. A
  host outage during leave, or a silenced phone, reaches nobody else.
- the **daily report posts into that same DM**, so alerts interleave with reports.

Switching to a team channel is a one-line change (`SLACK_CHANNEL=C…` in that env file)
and requires the bot to be invited there first, otherwise `chat.postMessage` answers
`not_in_channel`.

Without credentials it degrades silently to banner-only — Slack is never required.
Messages use Slack **mrkdwn** (single-asterisk bold, bare URLs) per the
design-system repo's `AGENTS.md`, and a bad token logs `WARN slack post failed:
{"ok":false,"error":"invalid_auth"}` rather than failing the run — Slack answers
HTTP 200 even on failure, so the response body is parsed instead of the status code.

Note this is a *different* Slack path from the `[ux_server] deploy 失败` messages you
already get: those come from the deploy-hook **on the host**. These come from your
laptop, which is the whole point — they still arrive when the host is gone.
- **No false alarms when the laptop is offline.** Every run first probes
  `https://1.1.1.1`. If that fails, the run logs `SKIP` and judges nothing — without
  this, closing the lid or changing networks would look exactly like a dead host.

## Coverage limits

Stated up front because the rest of this README could otherwise read as "you are
covered", and this monitor's whole reason for existing is that *silence looked like
health*. Same trap, one level up.

**It does not run while the laptop sleeps.** The agent is `StartInterval 900` +
`RunAtLoad`: macOS does not fire interval jobs during sleep, and launchd coalesces the
missed ones into a **single catch-up run on wake**. Consequences:

- a **night-time outage is not alerted while it happens** — you learn about it when you
  open the lid. The "reaches your phone at 3am" story the alerting section used to tell
  was **not true**, and has been corrected.
- an outage that starts and **self-heals inside the sleep window is invisible**: no
  probe ran, so no edge was ever observed. `monitor.log` will contain a gap, not an
  incident.
- the same applies to lid-closed, offline, and powered-off periods, and to the internet
  baseline check (which is what makes those cases `SKIP` rather than false alarms).

**It is also a single point of failure**: one laptop, one process. Nothing watches the
watcher.

Neither limit is fixed by anything in this repo — a heartbeat has to live somewhere that
does not sleep (a cheap always-on box, a cloud cron, or the NOC side) with this one kept
as a local convenience.

### That decision has been made: **keep as-is** (owner, 2026-08-11)

**This section is the single source of truth for that ruling and for when to revisit it.**
It used to say the decision was "tracked as backlog INFRA-F80" — that entry was closed and
deleted on 2026-08-13 precisely *because* the ruling had already been made, so nothing was
pending there. The narrative record lives in `docs/internal/STATUS-CHANGELOG.md`; the
operative part is here.

The owner ruled on both open questions and **changed neither**:

1. **The heartbeat is not moved to an always-on host** ("what we have is fine") — i.e. the
   trade-off stated above is *explicitly accepted*: outages during the working day are
   found within 15 minutes; outages outside it are found when the lid opens.
2. **Alerts stay in the owner's DM.** The two consequences recorded in the Slack section
   above (only the owner sees an alert; daily reports interleave into the same DM) are
   therefore **accepted properties, not unfixed defects**. Do not re-file them as bugs.

**Reopen conditions** — revisit only when one of these actually happens (absent one of
them, a fresh reading of this README should *not* re-raise the gap as a new finding):

- a night-time / sleep-window outage actually occurs **and causes real impact**;
- someone is about to use this monitor as an **on-call system**;
- what is monitored grows beyond "docs site + internal registry" to something carrying an
  **external SLA**;
- **someone other than the owner** needs to receive the alerts — that, and only that, is
  when the channel question reopens (and then see the invite-order requirement in the
  Slack section: change the env file *after* the bot is in the target channel, or
  `chat.postMessage` fails silently with `not_in_channel`).

**What this monitor does honestly deliver**: detection within 15 minutes *whenever this
Mac is awake*, which covers the working day — plus the stalled-deploy-chain check that
the host's own event-driven notifier structurally cannot do at all.

## Usage

```bash
./install.sh            # install/reinstall + start (runs once immediately)
./install.sh status     # loaded? current state? active alert? last checks
./install.sh uninstall  # stop and remove (keeps logs)

# run once by hand, without touching the installed state
STATE_DIR=/tmp/hm-once bash bin/check-host.sh && cat /tmp/hm-once/monitor.log
```

The agent is installed as a **symlink** into `~/Library/LaunchAgents/`, so editing
`launchd/com.tvu.host-monitor.plist` here is the only way to change the schedule —
there is no duplicate to drift. `install.sh` refuses to install if the plist's
`ProgramArguments` path no longer matches this checkout (i.e. after moving the repo).

## Verified behaviour (2026-07-29)

Exercised end-to-end rather than assumed, since the alert path is code that never
runs in normal operation:

- healthy run → all three checks `up` (`403` / `401` / `200`), `freshness age=0d`
- forced failure (targets repointed at a dead port) → alert fired, `ALERT.txt`
  written, `http=000` recorded
- second consecutive failure → **no** repeat alert (de-dup holds)
- recovery → recovery banner, `ALERT.txt` deleted, state back to `up`
- simulated offline laptop (baseline unreachable) → `SKIP`, no alert, no state
  written
- `osascript` exited cleanly on every alert (0 `WARN could not post` entries)

Installed agent confirmed live: `launchctl print` reports `run interval = 900
seconds`, `last exit code = 0`.

**Three bugs were found by this testing, all of which failed silently:**

1. BSD `awk` ignores `IGNORECASE` (a gawk extension), so `/^last-modified:/` never
   matched the real `Last-Modified:` header — the freshness check did nothing at all,
   and because its log line sat inside the success branch, it left no trace either.
2. `curl … || echo 000` concatenated into `http=000000`, since curl already prints
   `000` on connection failure *and* exits non-zero.
3. `launchctl list | grep -q "$LABEL"` under `set -o pipefail`: `grep -q` exits at the
   first match, `launchctl list` dies of SIGPIPE, and the non-zero pipeline made
   `status` report `loaded: no` for a correctly-registered agent. Now uses
   `launchctl print`.

Worth noting all three were in the *reporting and alerting* paths — the parts that
say whether things are fine. Testing only the happy path would have left a monitor
that looked installed and reported nothing.

## Verified behaviour (2026-07-30) — Slack end to end, and one more false-green

Slack was still unproven on 2026-07-29 (no channel). Now exercised against the real
credentials, not simulated:

- **negative control first**: scratch `STATE_DIR` (so no `.env`), `STALE_DAYS=0` → alert
  fired, `ALERT.txt` written, `freshness down` — and **0** `slack: sent` lines. That is
  the pre-fix state, and it proves the `slack_post` guard was the only thing muting it.
- **positive run** in the installed `STATE_DIR` → `slack: sent to <channel>`, and the
  message was then read back **independently** via `conversations.history` (matching
  `ts` and `bot_id`) rather than trusting the script's own log line. Recovery run →
  `RECOVERED`, `ALERT.txt` deleted, state back to `up`. (Ids intentionally not repeated
  here — see the Slack section.)
- **ingress swap regression** (4 states, scratch dirs): healthy → all `up`; proxy-layer
  failure injected via the new `BASE_HTTP` override → `gitea` **and** `registry` `down`
  while the `origin` diagnostic still read `403`, i.e. the log itself says "proxy layer,
  not Gitea"; a second identical run → no repeat alert (de-dup holds); recovery →
  `RECOVERED` + `ALERT.txt` removed.

**The false-green this found**: with `404` still accepted, the proxy-failure injection
left `registry` reporting `up` — the exact check whose whole purpose is "consumers can
install". Fixed by narrowing to `200 401` (see above).

`BASE_HTTP` / `ORIGIN_DIRECT` / `SITE` are now env-overridable *specifically* so failure
paths can be exercised on this file. The 2026-07-29 round had to copy the script to
`/tmp` and patch the copy — which tests the copy, not what `launchd` runs.

## If a banner never appears

`osascript` succeeding does not guarantee the banner is displayed — macOS may have
notifications suppressed for the delivering app. Check System Settings →
Notifications (look for **Script Editor** / your terminal), and confirm Do Not
Disturb / a Focus mode is not filtering it. `ALERT.txt` and `monitor.log` are the
reliable fallbacks, and `./install.sh status` reads both.

## Related

- `tvu-design-system/docs/DEPLOY.md` — the deploy chain, its three failure modes, and
  how to diagnose each
- `tvu-design-system/docs/internal/backlog.md` — **INFRA-F72** (the host-side deploy
  fix and the runner-springboard technique for reaching that host without SSH),
  **INFRA-F73** (the registry migration), **INFRA-F74** (TLS in front of Gitea — landed
  2026-07-30; that is why this monitor now probes `https://…/gitea/`)