One of my jobs is keeping client workspaces current. When we improve the shared tooling, an update pipeline detects the change and installs it automatically.

In March, the new pipeline worked perfectly in testing, then failed on all 4 existing workspaces at once.

The reason is almost a riddle. The thing doing the delivering was the thing being delivered.

The old courier didn’t know about parcels

The pipeline works like a courier. A small piece of code runs at the start of every session. It checks whether the installed tooling has drifted from the reference copy, and installs whatever’s new.

The rollout plan was clean. Push the new files, let each workspace sync, let the courier notice the drift and install.

Except the courier itself was the thing being upgraded:

  • The old version, about 2 KB of code, only knew how to poll a version number.
  • The new version, about 13 KB, knew how to detect drift, classify it, and auto-install.

The new courier arrived at every workspace and sat in the reference location, waiting to be installed. The installer on duty was an old courier that had never heard of installing things.

Testing never caught it. Every test environment already had the new courier, and fresh workspaces got it at packaging time. The simulations all began, in effect, “once the new system is running.”

Nobody simulated the one state every real workspace was actually in: old code live, new code waiting.

Bootstrap boundaries

The one-time fix was manual: copy the new courier into place by hand, once per workspace. After that, every later update flowed through automatically. The thing doing the delivering finally knew how.

The lasting fix was a question, added permanently to our rollout process. Can the target environment actually execute this change, or does the change depend on infrastructure that doesn’t exist there yet?

Every self-maintaining system has a bootstrap boundary. On one side: things the system can update. On the other: things the system needs in order to run. Cross that line and self-deployment quietly stops working:

  • Package managers can’t install their own next version through the dependency resolution they’re shipping.
  • CI/CD pipelines can’t deploy changes to their own deployment infrastructure.
  • Migration tools can’t migrate the table that tracks migrations.
  • Auto-updaters need a separate path for updating the updater.

Before you ship a change to the shipper

The check we run now, liftable for any self-updating system:

  1. List every component the update path needs in order to function.
  2. Ask whether the thing being changed is on that list.
  3. If yes, design the bootstrap step explicitly - it will not deploy itself.
  4. Test on an environment running the OLD version, never just the new one.

The lesson hangs on my wall in one line: a pipeline that maintains itself still can’t be born by itself.