Where the Failure Began: Attribution Before Recovery

The reported exception shows where the workload stopped, not where the failure began. NVIDIA states the problem in one sentence at the top of the November 2 post, and the sentence deserves framing on the wall of every operations room. In distributed Artificial Intelligence workloads, a single GPU or fabric event can surface as collective communication errors on other ranks, scheduler changes and thousands of downstream messages. Every log tells the truth about a symptom. None of the logs, read alone, tells the truth about the cause.

On November 2, 2026, NVIDIA open sourced Attribution, the workload-centric failure-attribution system used across NVIDIA AI infrastructure. NVIDIA reports that the system has run in production for years and analyzes millions of workloads each week across hundreds of thousands of GPUs. Those scale figures are vendor-reported: context for the design, not a benchmark for any single cluster. A hard-won operations discipline, previously internal, now ships as inspectable code and rules.

01 · The announcement

Attribution ships as two deployable components around one core engine. The Attribution Service wraps the engine in an HTTP interface and analyzes one workload attempt at a time. The Attribution Blueprint, a Python reference orchestrator, gathers workload context and evidence, calls the Service, enriches the verdict and writes the result. Connectors cover Slurm, Kueue and static inputs, plus Loki, local files, Kubernetes pod logs and PromQL metrics.

The path runs in four stages. The Blueprint scopes the attempt from scheduler identifiers or an explicit execution window and node allocation. Connectors collect workload-scoped observations from application logs, system logs, GPU driver and fabric messages, scheduler history and node and network metrics. The engine extracts events and correlates them, using deterministic rules to separate downstream symptoms from the event that caused the failure. The result returns as structure: outcome, attributed domain, symptoms, affected nodes and collector coverage.

02 · Why it matters

Failure at cluster scale is a counting problem before the counting becomes a craft problem. NVIDIA illustrates the arithmetic with a study cited in the companion Recovery post: across an 11-month study of a very large GPU fleet, hardware failures interrupted fewer than one percent of jobs, yet those jobs accounted for a far larger share of total GPU runtime. Hold the figures as NVIDIA reports them from the cited study, not as universal rates. Rare faults, multiplied by enough GPUs and hours, become a standing cost.

The production colour points the same way. In one large RDMA-backed cluster, the system correlated port errors, connection resets and scheduler events into patterns around specific host and switch ports, guiding fabric repairs and congestion tuning. A diagnosis that names a node and a cause turns an outage review into a repair order. A diagnosis that lists every error ever printed turns the same review into archaeology.

The reported exception shows where the workload stopped, not where the failure began.

03 · The operations angle: scope, correlate, then act

The worked example rewards a slow read. Application logs for a multi-rank training workload contain three different collective communication errors. System logs on one node carry a GPU hard fault whose error words decode to a signal integrity issue on the interconnect. Early filtering decodes that evidence while leaving the communication errors unchanged. Final refinement surfaces the signal integrity issue on the named node as the root cause, demotes the other symptoms, and keeps the communication errors in the record as interesting evidence. The workload is reported as a hardware failure, and the responsible node is named.

The companion release, NVIDIA Recovery, shows what a trustworthy verdict makes possible. The loop runs in four stages: detect crashes, hangs and stragglers from logs and scheduler signals; attribute the interruption, with an optional Attribution verdict distinguishing hardware faults, system software issues and application bugs; isolate implicated nodes, with a watchdog safety net against indefinite waits; then restart from checkpoint on healthy capacity, or stop retries that have stopped being productive. Every recommendation is recorded before action, in an audit trail holding the pattern, the action, the timestamp, the workload and the verdict when one was returned.

Context engineering at Ovia since December 2022 rests on the same discipline. Evidence gets scoped to the work before judgement begins. Signal gets separated from the noise the signal caused. Decisions carry records a later reader can audit. The field years behind the practice — technology coordination at Staywired from 2014 to 2022 and project facilitation at Beanfield Metroconnect from 2011 to 2013 — taught the physical version: the loudest alarm in the building is seldom the fault in the building. Certification in NVIDIA AI Operations adds the cluster version. Attribution is that craft, written down as rules.

04 · Caveats and fine print

NVIDIA states the central limit without decoration: the quality of an attribution depends on the quality and completeness of the underlying telemetry. When several causes fit the evidence, the attributed domain is reported as ambiguous. When evidence runs short, the honest output is an unattributed result rather than a confident guess. That restraint is a feature worth copying into diagnostic systems, because a forced verdict repairs healthy hardware and leaves the real fault in service.

Two governance details deserve attention. Attribution rules ship with the Service as committed, tested code; rules are not uploaded dynamically through the HTTP interface, On the Recovery side, Slurm carries the supported path, including Slurm on Kubernetes through the Slinky operator, while native Kubernetes recovery sits on the roadmap. The Recovery figures NVIDIA reports come from one cluster, with latencies measured to the automated action; scheduling and checkpoint restoration decide when training resumes. New deployments start in monitor mode: validate the advice first, then grant action.

05 · What to watch

Watch the rule catalog first. The open design invites teams to encode a failure pattern once and apply the pattern across workloads and clusters. Watch the integrations second: Recovery already accepts verdicts from a custom attribution service, an early partner contributed a connector for an alternative log backend, and downstream incident and fleet systems can consume the structured verdict directly. Watch, finally, for the roadmap items NVIDIA names openly: recovery for job arrays and native Kubernetes recovery.

The deeper pattern is the one to keep. Diagnosis is becoming a contract: scoped evidence in, structured verdict out, uncertainty stated, action recorded in an audit trail. Operations gain a memory — every failure leaves behind a named cause, a named node and a rule that recognises the pattern next time. That is how a cascade becomes a curriculum, and how the next failure begins, already half explained, where the last failure ended.