The other day we added a new server node to a virtualization platform running in production. We ran the job as a division of labor with an AI agent (Claude), so here is the record.

The short version: the split worked out remarkably cleanly. But this is not a story about handing everything to an AI and having it all come out fine. If anything it is the opposite: every call that prevented a serious incident came from the human side. That part is in here too.

The split in practice #

Physical to the human, logical to the AI. Put crudely, that is all there is to it. But as described below, the side that knew what was actually dangerous was the human side.

Where the AI paid off: legwork, repetition, cross-checking #

The AI was clearly strong at the work that is boring, high in volume, and painful to get wrong.

1. Exact version parity #

The existing cluster is run with every node held at strictly identical versions. The new node was fresh out of an OS install, with packages several generations behind.

Simply updating to the latest would have put it ahead of the existing cluster, which breaks that policy. But pinning a specific version meant dependencies could not be resolved and the install failed.

The AI isolated the cause from the error messages (unless the related packages are all named together, the old versions stay behind and collide; and it had been pulling in a server-side component that had no business being installed at this stage, which jammed the whole thing), and landed on a version set identical to the existing nodes down to the last digit. A human doing this kind of trial and error runs out of concentration long before it ends.

2. Diff verification across every node #

After the expansion we checked, one machine at a time, whether the shared storage looked identical from every node. The more machines there are, the more this becomes a step you want to do by hand and settle with "probably fine".

The AI runs the same command against every machine and lays the differences out in a table, and it does it without ever getting bored. The result was that we confirmed the new node had ended up in exactly the same state as the existing ones by measurement, not by assumption.

3. Evidence matched at "both ends" #

For link aggregation (LACP) we did not only check the server side; we matched it against the state on the switch side too. In particular, we went as far as verifying that the peer identifier visible from the server matched the switch we actually intended. That is proof not that "the link came up", but that "it is connected to the intended peer, with the intended configuration".

The unglamorous but dangerous things the AI found #

During the work the AI turned up several problems that nobody had asked it to look for.

Leftovers from a previous OS in the boot order #

On one node, the top of the UEFI boot order still held an entry for a different OS that had been installed at some point. Nothing was actually on disk behind it, so it could not boot; behind that came a row of network boot entries, and the correct entry was seventh.

Reboot it in that state and it would hunt for an OS that was not there, sit through network boot timeouts one after another, and with bad luck, boot something we never intended. Catching this before any reboot was a big deal.

A cascading clock failure (this one is the most fun) #

On another node, package updates all failed. The error: "the repository metadata is not valid yet (901 days to go)".

Tracing it back, the chain went like this.

1. The hardware clock (RTC) read 1998 (a dead battery is the likely cause)
2. The OS judged that to be bogus and pushed the clock forward to its own build date (about 2.5 years ago)
3. Time sync was configured to use NTP servers on the internet, but the DNS configuration was broken, so those servers could not be resolved and there were zero sync sources
4. → the clock stayed 2.5 years in the past → the package manager treated the repository as carrying a signature from the future and rejected all of it

The real cause of "packages will not install" turned out to be "DNS is broken" — four hops down the chain. Fixing DNS and restarting time sync cleared it instantly.

And left alone, this would have come back to bite us once the node joined the cluster. Distributed systems are sensitive to clock skew between nodes, and putting a node that is 2.5 years off into a production cluster is out of the question.

A setting that "failed silently" #

The mechanism that applies network performance tuning had run at boot before its target existed, done nothing, and exited successfully. No error is raised. In the log it is a success.

The AI predicted this from the order of operations, went and checked, found that it really had not been applied, and re-ran it. "Reports success but has actually done nothing" is the class of problem a human eyeballing things is most likely to miss.

Where the human mattered: this is the real point #

Here is the heart of it. Both of the calls that prevented a catastrophic incident this time came from the human side.

1. "The file placement for the integration is missing" #

The AI had organized the task list, but it had filed the file placement needed for integration with an external system under "the phase we do at the very end". In reality, out of everything in that phase, that one file placement was fine to do earlier — and in fact should have been done earlier.

It took the human saying "you have missed something" for the AI to notice its own misclassification. Without that, we would have tripped over it later in the process.

2. "The storage-side registration comes first. The nodes will end up seeing different things" #

This is the big one.

As a pre-join check, the AI lined up versions, networking, clocks and so on, and declared the node ready. But registering access permissions on the storage array was not on that list.

The human pointed out: "join it before that registration and the shared storage will look different from one node to the next, which can cause serious inconsistency".

When the AI went looking, there was an internal record of exactly this happening before. One machine could see the storage and the rest could not, and rebooting one node dragged unrelated nodes into rebooting with it.

Without that warning, the AI would have laid out a beautifully tidy set of verification results and driven us straight into the incident.

In the end we registered the access permissions first, measured non-destructively before joining that "the storage list visible from the new node matches the existing nodes exactly", and only then joined it. It finished with zero incidents.

What these two have in common is a "sense of danger born of experience". The AI draws a coherent conclusion from the information in front of it, but it cannot notice that an item is simply not on the list in the first place. The one who knew what belonged on the list was the human who had been burned before.

What the AI got wrong (written up without hiding any of it) #

In fairness, here are the AI's failures too. You should not trust a case study that leaves this part out.

• It wrote the wrong order into the runbook and carried it straight onto real hardware. The order for applying the settings was written backwards, and we only found out when the real machine threw an error. Luckily it was a step with no impact on traffic, so no actual harm
• It thought it had verified something it had not. The pre-flight check on a setting was returning an error, but the script was written badly and printed "OK". It owned up later and fixed it
• It made noise about something that was not a problem. It compared against one machine only and reported that the settings did not match. Checking every machine showed that the new node was in fact the correct one
• It visibly stopped making sense and a human stopped it. Mid-work, the AI started reacting to a topic unrelated to the context and going off on tangents. The human sensed something was wrong, halted all work, checked the state, and only then resumed

That last one matters most. An AI cannot detect its own malfunction. Production work always needs a human who can call a halt the moment something feels off.

The lesson: how to design the division of labor #

Sorting out why this worked, it comes down to the following.

What was worth handing to the AI #

• Repetition, cross-checking, checking every single item (the steps a human is tempted to skip with "probably fine")
• Dependency resolution that takes trial and error
• Verification that leaves a paper trail as it goes
• Folding the work back into the runbook (written straight after doing it, so it never goes stale)

What the human has to keep #

• Deciding what goes on the list (the AI cannot notice an item that is not on it)
• Flagging risk from the memory of past incidents
• The call to stop
• Physical work (obviously)

And what worked for both sides was the same approach: "run small, measure every time". One machine at a time, one step at a time, verified by measurement after every change. Because that pattern was in place, both the AI's mistakes and the human's oversights were caught while the wound was still shallow.

In closing #

Say "the AI built the infrastructure" and people may picture a human sitting there issuing instructions and nothing more. That is not how it went.

The AI is overwhelmingly strong in the territory where humans lose focus. The human is decisively strong in the territory the AI structurally cannot see. When those two mesh, you get a quality that neither reaches alone.

And if you are going to put an AI into production, you need a human who can stop it. That this worked here is the single thing we most want to share from this case.