An incident orchestrator is a checkpointed graph, not a script

Why not a script

The scenario for handling a backbone-link incident looks linear right up until you try to write it. Monitoring opens a ticket; you parse it, connect to both ends of the link, read optics and interface config, draw a conclusion, deactivate the port, file a task for the field team - and wait. That wait may be an hour or three days, until a crew reaches the splice closure.

A script does not survive that. It dies on the next service deploy, and with it dies the knowledge that a port is already deactivated and someone is on the way to fix it.

So the scenario is built as a state graph on LangGraph, with graph state stored in PostgreSQL checkpoints. A restart is not a loss of context but a pause: the process resumes at exactly the node where it stopped.

Decision 1: the LLM never touches devices

The temptation to give the model CLI access is enormous, and it is precisely what you must not do. The split is strict:

• deterministic code - NETCONF/PyEZ connections, output parsing, config application;
• the model - text only, plus a compact JSON digest of the diagnostics, from which it writes a conclusion for the engineer.

Everything that changes the network goes through code you can read and cover with tests. The model shapes the wording and the "looks like a fiber break" verdict - not the commit.

There is a pleasant side effect: a digest instead of raw command dumps is also an order of magnitude fewer tokens.

Decision 2: commit confirmed instead of hope

Junos can roll a config back on its own:

# the rollback timer lives on the device itself
cu.commit(confirm=10, comment="incident <ticket>: deactivate port")
# ... an engineer confirms in the UI ...
cu.commit()  # finalize; without this the device reverts by itself

This changes the failure model. If the orchestrator crashes, loses the network or simply gets it wrong, the port returns to its original state without us, because the timer lives on the device, not in our process. The human stays in the loop: confirming the change is their action, not a flag in a config file.

Decision 3: dry-run first

The first stage of the project runs deliberately without a single real commit. Instead of applying config, the orchestrator computes a diff and logs what it would have done:

WOULD COMMIT on <device-a>:
[edit interfaces xe-0/0/0]
+   inactive: gigether-options

Boring - which is exactly why it works. Dry-run is where you find out how the parser trips over non-standard ticket descriptions, where diagnostics produce a false verdict, and which "both ends of the link" pairs are missing from inventory. Better to learn that from a log than from an incident.

What is left out

The ticket description format is not a contract - it comes from a monitoring template, and it changes. The parser is robust to the variants seen so far, but when the interface list comes back empty it stops and calls a human instead of guessing. That is probably the core principle of the whole system: automation is allowed to refuse to act.