Why an incident orchestrator should be built as a checkpointed graph

Why not a script

Handling a backbone-link incident looks linear until you try to write it. Monitoring opens a ticket, and then you need to parse the description, connect to both ends of the link, collect optical readings and the interface configuration, reach a conclusion, deactivate the port, file a task for the field team and wait. The wait can last an hour or three days, until the crew reaches the splice closure.

A script cannot survive that. It stops at the first service deployment, and with it goes the knowledge that the port is already deactivated and a crew is on its way.

That is why the scenario is built as a state graph on LangGraph, with graph state stored in PostgreSQL checkpoints. After a service restart the context is preserved, and the process resumes at the node where it stopped.

Decision 1: the LLM never connects to devices

Giving the model CLI access is very tempting, and it is exactly what should be avoided. Responsibilities in the system are strictly separated:

• deterministic code connects over NETCONF/PyEZ, parses output and applies configuration;
• the model works only with text and a compact JSON digest of the diagnostics, from which it writes a conclusion for the engineer.

Every change to the network goes through code that can be read and covered by tests. The model shapes the wording and conclusions such as "looks like a fibre cut", but it never runs commit.

This approach has a pleasant side effect: the digest takes several times fewer tokens than raw command output.

Decision 2: commit confirmed

Junos can roll back configuration on its own:

# the rollback timer lives on the device itself
cu.commit(confirm=10, comment="incident <ticket>: deactivate port")
# ... an engineer confirms in the UI ...
cu.commit()  # finalize; without this the device reverts by itself

This changes how the system behaves on failure. If the orchestrator crashes, loses connectivity or makes a mistake, the port returns to its original state without our involvement, because the timer runs on the device itself. A human stays in the loop, since confirming the change is their action.

Decision 3: dry-run first

The first stage of the project deliberately runs without a single real commit. Instead of applying configuration, the orchestrator computes a diff and logs what it would have done:

WOULD COMMIT on <device-a>:
[edit interfaces xe-0/0/0]
+   inactive: gigether-options

This stage looks boring, but it is very useful. Running in dry-run shows where the parser stumbles over non-standard ticket descriptions, where diagnostics reach a wrong conclusion and which link-end pairs are missing from the inventory. All of this is better learned from a log than from an incident.

What is still open

The ticket description format is not fixed by a contract. It is generated by a template in the monitoring system, and that template changes. The parser handles every variant seen so far, and when the interface list comes back empty it stops and calls a human. This is probably the main principle of the whole system: automation is allowed to refuse to act.