RTNL responsiveness watchdog

rtnl-watchdog measures responsiveness of a small request through the global RTNL path. It runs independently of oomd and does not identify a lock holder, select containers or take container actions. See the oomd reference for the optional consumer policy and guarded actions.

The service is disabled by default. Enable its standalone diagnostics with:

{
  services.rtnl-watchdog.enable = true;
}

The root-owned service uses one observer, one persistent private probe helper and one bounded output worker. The helper performs a non-dump RTM_GETLINK request with no interface selector or attributes. On the supported kernel this path takes global RTNL and returns the deliberately expected EINVAL before device lookup and response filling. Only a kernel response matching the request sequence, type and expected error is a valid completion. A successful send alone is insufficient.

Both binaries are written in Rust, with Linux syscall bindings isolated in a small module. Before starting worker threads, the observer marks descriptors 3 and above close-on-exec. Startup fails if it cannot establish that policy. Rust ownership does not guarantee progress in a kernel call.

The private helper receives duplex IPC on stdin (FD 0) and the inherited lifetime lock on stdout (FD 1). It never writes or logs to stdout; stderr goes to /dev/null. Closing a local descriptor, receiving EOF or losing the observer does not cancel a pending request.

The helper must share the host init network namespace and cannot switch namespaces. The module runs in the ordinary host service context; standalone use also requires that context. Generic socket, acknowledgement, scheduling and unlock work are included in the duration. This measures request responsiveness rather than exact mutex sleep time. Kernel support must be verified with the selected kernel; an unexpected response is an error. The expected error alone does not prove global-lock coverage on an unexamined kernel version.

Settings

All options below are under services.rtnl-watchdog.settings:

Option Default Supported range
intervalMs 1000 100 to 60000 ms
thresholdMs 1000 100 to 60000 ms; at least tickMs
tickMs 250 50 to 1000 ms
errorBackoffMs 5000 100 to 3600000 ms
metricsFile /run/metrics/rtnl-watchdog.prom Absolute file path

The observer admits no backlog or overlapping request. The next probe waits at least intervalMs after completion; completed errors wait the larger of that interval and errorBackoffMs. While an announced request remains pending, its age crossing thresholdMs produces distress. The first probe can become distressed before any successful response. A missing start acknowledgement remains unknown, regardless of dispatch age. A matched valid completion clears the current distress immediately and retains its full duration.

The CLI mirrors these settings with --interval-ms, --threshold-ms, --tick-ms, --error-backoff-ms and --metrics-file. Standalone fixtures can also set --state-file and --lock-file; the module fixes those paths below. Paths must be absolute and normalized, with distinct state, metrics and lock files in trusted root-owned directories. --help and --version do not probe. services.rtnl-watchdog.package selects a package containing the observer and its private helper; the helper is resolved from that package, never from PATH or a configurable backend.

State and freshness

The authoritative consumer file is /run/rtnl-watchdog.json, atomically replaced with root ownership and mode 0600, limited to 4096 bytes. Schema 1 fields are schema, boot_id, epoch, sequence, observed_monotonic_ns, probe_sequence, state, dispatch_monotonic_ns, started_monotonic_ns, completed_monotonic_ns, last_duration_ns, expected_errno and error_code. Unset timestamps/durations are null. Historical last valid duration remains available while a subsequent request is pending. expected_errno is 22. error_code is 0 for none, 1 for namespace, 2 for socket, 3 for send, 4 for receive, 5 for reply, 6 for IPC and 7 for helper failure. Status logs include observer/helper PIDs, epoch and reason.

States are starting, pending, distressed, healthy and error. distressed requires a received start acknowledgement and an overdue pending request; the acknowledgement does not prove the syscall has reached RTNL. healthy requires the matched expected response. Namespace/protocol/helper errors remain unknown. Consumers validate root ownership/mode, framing, boot/epoch/ordered sequence and monotonic freshness. Oomd requires two ordered fresh frames to bind a new epoch and expires evidence after one second. Old, future or missing state cannot authorize its watchdog override.

Prometheus output

Use the existing node-exporter textfile collector for the configured directory. The watchdog opens no HTTP listener. Metrics use rtnl_watchdog_: available, distress, pending_age_seconds, last_probe_duration_seconds, last_success_timestamp_seconds, snapshot_timestamp_seconds, errors_total and output_errors_total. Unknown distress is omitted; last duration/success are omitted until a valid completion. Check freshness and availability before interpreting them.

output_errors_total counts failures to render or write state, metrics or status output, including output rejected by a size limit. Replacing an older queued snapshot or status event with the newest one does not increment it. The counter resets when the observer restarts. Alert on new failures and stale snapshot_timestamp_seconds values rather than a positive lifetime total: a past error can remain visible after publication has recovered. A failed metrics write can only be reported through a later successful publication.

Older implementations also count queued-output replacements as output errors. During a mixed-version rollout, interpret this counter per node with that difference in mind.

response_duration_seconds is a histogram of valid completed responses. Its fixed bucket limits are 0.001, 0.01, 0.1, 0.25, 0.5, 0.75, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 25, 30, 40, 50, 60, 90, 120, 300, 600 and +Inf seconds, with count and sum series. Buckets are cumulative: each le bucket counts responses less than or equal to its limit. It records only matched expected-EINVAL completions. overdue_probes_total counts each announced probe once at an observed pending threshold crossing or a late completion, including timed errors; missing-start dispatches are excluded. These aggregates retain short stalls between scrapes even when the last-duration gauge is later overwritten. They persist across helper recreation and reset with the observer process, as ordinary Prometheus process counters do. A restart before a scrape can lose measurements not yet collected. Only histogram le labels are used; detailed bounded error reasons belong in logs.

When aggregating histograms across nodes, use matching bucket boundaries. During a rollout with different bucket sets, query each node separately or restrict aggregation to the boundaries shared by all selected nodes.

State/metric output uses a fixed latest-frame handoff. Coalesced frames retain observer-owned histogram/counters. A full stdout pipe drops bounded status events without stopping local file output. A blocked filesystem writer can leave both files stale while the observer/helper continue; it is never replaced by an unlimited worker pool. Host scheduler or memory stalls can also delay observation. There is no guaranteed alarm deadline during a global stall.

Stop and recovery

An exclusive lock at /run/rtnl-watchdog.lock is inherited by the helper across exec. The module uses runit's process-only stop mode. If the observer dies while a helper is blocked, that helper retains the lock and a replacement observer fails visibly instead of launching another probe. The lock file is never unlinked or recreated as a deadline or shutdown mechanism.

Normal shutdown stops admissions and closes IPC; the helper exits after its outstanding call returns and detects the closed connection. No cancellation is claimed. After a known reaped helper failure, recreation waits the error backoff and starts a new epoch. A child with uncertain effects is never replaced. Clean observer shutdown removes only its own state/metrics after the writer exits. A crash or stalled writer can leave stale output; consumers must enforce freshness. Do not delete the lifetime lock or force a second probe during recovery. Disabling the service changes no persistent schema or container lifecycle, and oomd consumer policy remains separately controlled.