Container OOM daemon

osctl-oomd checks container memory, CPU and load every 15 seconds by default. Its host-load gate compares the current load with a configured multiple of the historical median. Candidates must meet the memory threshold, meet the CPU threshold when a CPU limit is set, and fall within the configured load ranking. The synthetic host row participates in that ranking.

At the default interval, eight hits request a restart and forty request a stop. A qualifying observation adds one to a score; a miss subtracts one, with a floor of zero. An inactive host gate leaves scores unchanged.

Enable the RTNL watchdog gate

The watchdog is a separate service. It can export Prometheus metrics without activating oomd. See the RTNL watchdog guide for the probe, metrics and lifetime-lock recovery rules.

To let oomd react to watchdog distress:

{
  osctl.oomd = {
    enable = true;
    settings.rtnl_watchdog.enable = true;
  };
}

osctl.oomd.settings.rtnl_watchdog.enable defaults to false. With oomd enabled, this option also enables services.rtnl-watchdog. Oomd uses ordinary JSON ct top with network, ZFS, iostat and process collection disabled. A blocked required read can still prevent a fresh sample. Oomd reads the local watchdog state; its reaction does not depend on a Prometheus scrape or network request.

Eligibility samples from ct top

osctl -j ct top --rate 15 --no-netif --no-zfs --no-iostat --no-processes

Ordinary ct top defaults to a one-second interval. Oomd supplies its configured observation interval, normally 15 seconds. The same Top model serves JSON and the interactive display. Network and ZFS collection are enabled by default; --no-netif skips interface inventory, events and statistics, while --no-zfs skips objset, ARC and iostat reads. The disabled JSON fields are null.

Top collects cgroup v1/v2 CPU and memory counters and limits, host CPU/memory/load, and per-container load. It validates running inventory and init PID/start identity around acquisition. Memory values are bytes; CPU usage keeps Top's percentage-of-one-CPU convention and existing limit units. Unlimited limits are null. The synthetic [host] row has a null pool and run identity and participates in the full load ranking; it is never a container action target.

The enabled watchdog consumer requires top_schema: 1 and effective collectors: {netif: false, zfs: false, iostat: false}. Missing, malformed or unexpected collector settings cannot authorize actions. This requirement belongs to the consumer; ordinary Top can use any supported collector settings. Frames include sampler-process epoch, increasing sequence, complete and invalid_reason. Running container rows retain pool, id, loadavg, memory, memory_limit, cpu_usage and cpu_limit, with init_pid, init_start_time_ticks, valid and invalid_reason identifying usable runs.

sample_started_monotonic_ns and sample_completed_monotonic_ns enclose the current acquisition and projection. measurement_window_started_monotonic_ns is the preceding coherent same-run raw sweep's start; measurement_window_completed_monotonic_ns is the current sweep's completion. These host-monotonic timestamps enclose the CPU reads rather than timestamping each counter read exactly. Serialization and redraw do not advance the sequence or refresh the sample's age.

Oomd checks current-sweep duration and age since completion separately. Each is limited by observation_max_age_seconds, which defaults to five seconds. The CPU envelope may span the observation interval, but must fit within that interval plus the freshness limit: twenty seconds with both defaults. Consecutive complete frames must link their CPU window to the preceding sweep. A stretched or inconsistent window is rejected even when its latest completion is recent.

Missing or inconsistent core input makes a snapshot incomplete. A failed sweep discards its baseline. A coherent raw sweep without a compatible baseline seeds the next delta and remains incomplete; counter resets and run changes also require a fresh baseline. Setup takes an unpublished seed, so the first visible complete frame can use that seed. Disabled optional collectors do not invalidate coherent core data, and unavailable optional values are null.

Consumers must reject incomplete or invalid rows before interpreting eligibility. Required read failures cannot certify zero activity. A blocked required read prevents fresh publication; no replacement reader is launched.

Scoring during watchdog distress

Fresh watchdog distress opens the host gate independently of loadavg. It does not identify the responsible container. The existing memory, CPU and ranking checks still apply.

Each distress episode starts with cleared restart evidence. The episode restart threshold defaults to two. Only distinct qualifying observations whose sweeps began after episode entry add hits. Misses subtract one from restart evidence; watchdog heartbeats and duplicate samples do not add hits. The original loadavg-gated policy continues to update the stop score, including its default forty-hit threshold. Watchdog-only observations never increase or decrease it.

A fresh healthy completion, stale or erroneous watchdog state, or an observer epoch change removes the override and clears accelerated restart evidence. Each new watchdog epoch needs two ordered fresh frames before it can open the gate. Invalid or incomplete samples, missing sequence numbers and sampler replacement also clear accelerated evidence.

The sample freshness limit applies to acquisition and action dispatch. A hit survives the normal fifteen-second observation interval. If the next valid observation does not arrive within the interval plus the freshness limit, accelerated restart evidence clears without creating a miss. Same-run ordinary-mode scores survive invalid gaps and sampler replacement; dispatch still requires a fresh, complete current-run sample.

Settings under osctl.oomd.settings.rtnl_watchdog:

Setting Default Meaning
enable false Enable the additional host gate and guarded action path
state_file /run/rtnl-watchdog.json Root-owned schema 1 state file
max_age_seconds 1 Maximum watchdog frame age
restart_hits 2 Episode restart threshold, at least two
observation_max_age_seconds 5 Maximum sweep duration and sample age at dispatch
action_workers 2 Physical action slots, restricted to one or two

Guarded actions

One controller owns scores and action reservations. Two command slots allow another eligible container to proceed while one action blocks. A busy slot does not accumulate authorized commands. A subsequent fresh observation must pass the host gate, eligibility, age and run checks again. Each container has at most one action in flight. A successful restart retains its stop history. Completion or rejection requires new raw evidence before another action.

osctl ct oomd-restart and osctl ct oomd-stop require both --expected-init-pid and --expected-init-start-time. The latter uses raw process start ticks. The daemon checks them under the container manipulation lock before entering the existing force-kill lifecycle. A request that waits for the lock checks the run after acquiring it. Ordinary stop and restart commands are unchanged. Old daemons reject the new command names; there is no fallback to an ordinary command.

After validation the daemon emits OOMd expected run accepted pid=… start=…. Oomd sends the same guarded daemon requests through the existing client API. Only a definitive terminal daemon response establishes completion. EOF, transport errors and protocol uncertainty leave the outcome unknown. Guard acceptance, payload EXIT and command completion are separate events.

NFS cancellation, freezing, force-kill and cleanup retain their existing order. Shared kernel or NFS locks can block both slots indefinitely. A deadline does not free a slot or establish cancellation. An unknown command outcome keeps its slot and reservation held and does not trigger automatic retries. Shutdown stops new actions and waits for outstanding work.

Event broadcast and policy logging use a separate worker with a sixteen-event queue. Notification failure does not retry an action. Queue overflow and failure appear as osctl_oomd_notification_drops and osctl_oomd_notification_failures; uncertain effects appear as osctl_oomd_unknown_actions. The existing textfile exporter publishes these metrics. A blocked notification worker cannot retain a completed action slot.

Compatibility and rollback

Install the matching Top sampler, watchdog and guarded osctld/osctl commands before enabling the option. Old or incomplete sampler output cannot authorize feature actions. An older replacement daemon rejects guarded actions by command name.

Disabling the option restores the original policy path and thresholds. It requires no persistent score or database migration. The host-load history keeps its existing format and cadence. Wait for outstanding actions to finish before replacing the service: changing settings does not cancel a blocked command. Follow the watchdog guide's lifetime-lock recovery rules when disabling or replacing that service.