At my day job I ship a large Windows desktop application. A while back I wrote a console app that takes it from source to a signed, published installer. It fetches the source, stamps a version, compiles, packages, signs, uploads, updates the manifest the in-product updater reads, and posts a card to team chat. If a step looks wrong it stops. That replaced an afternoon of clicking through tools with one command.
It also runs on a schedule. Every night it builds each active branch and product version and pushes the results to their test channels. Nobody touches the machine and testers have fresh installers in the morning. That has been the case for a while now.
More recently I put an agent in front of it. The agent has the build machine as its working directory and can do anything the nightly can, plus the things the nightly was never set up for: build one branch with a particular configuration, run one test against last night's output, work out why something failed. I ask in plain English. It spawns sub-agents for the long jobs, runs the same PowerShell scripts the nightly runs, checks the result, and reads the logs before I get to them. The console app is old news. The agent is what I want to write about. Names, hosts and anything that would identify the product are left out.
What the agent inherited
The pipeline was already built to run without anyone watching, which is most of why an agent can drive it. A scheduler starts it at night, once per branch and version. A small web form starts it on demand. Both call the same executable with flags. Every stage writes a log line. Three checks exist because each of the failures they catch has shipped: a compile that reported success but produced nothing new, files an upgrade skipped because their internal version never changed, and an upload that landed partial while the filename said it was fine. The pipeline now checks a sentinel binary's timestamp after compiling, stamps every shipped file with a sentinel-high version so an upgrade always overwrites, and compares byte counts after uploading instead of trusting the name.
$ buildbot --channel beta[00:00] fetch pull source, wait for referenced dependencies to settle[09:40] stamp version 26.1.418 written to the shared header[09:41] compile product solution[54:12] prove sentinel binary written 31s ago, ok[54:13] package inner package, then bundle with prerequisites[61:02] force shipped files stamped so upgrades cannot skip them[61:40] sign package ok, bundle ok, timestamped[62:10] publish streamed upload complete, byte count matches[66:55] announce beta manifest refreshed, card posted to team chat[66:56] advance next build number is 419A full build takes a bit over an hour, most of it compiling. The output is a single installer over three gigabytes. It carries the product, a large third-party platform the product plugs into, and around a dozen prerequisites, each with a detection rule so it gets skipped when already installed. None of this runs on a hosted CI service. The licensed toolchain and the signing certificate live on one machine, so the build lives there too. I did not plan it with agents in mind, but a local, scriptable, logged build is exactly what an agent needs.
Talking to the pipeline
The agent's session sits in the project folder on the build machine. The nightly does not need it. I use it for the requests the nightly does not cover. Build the maintenance branch with the hardware interface left out and tell me if the bundle still fits. Rerun the install log check on last night's beta, language pack only. Cut a beta from current source and confirm it landed. The agent turns each one into flags on the executable, watches the log, and afterwards checks the signature, the upload size and the published manifest. I do not have to remember the flags for the odd cases anymore, and I do not have to be at the machine. I have kicked off builds from my phone.
> build the maintenance branch as a beta, then check it landed spawning build agent buildbot --branch maint-25 --channel beta spawning verify agent waits on publish, then checks signature, size, manifest [67 min] build 26.1.418 published to beta verify: signature valid, timestamped; remote size matches local; manifest points at 418 commits since 417: 14, attached to the cardOne sub-agent per job
Long jobs get their own session. The main agent spawns one sub-agent to run the build and another to verify it, and each reports back when it finishes. The main conversation never blocks for an hour. The sub-agents absorb the noise: forty-five minutes of compiler output, the installer's verbose log, upload progress. The main agent gets a short result back and keeps track of what I originally asked for. If I ask for three configurations it spawns three builds, so "does this change break any supported version" gets answered in one evening.
New jobs get added the same way, since a sub-agent is a prompt and a working directory. Localization is the one in progress right now. A sub-agent goes through the resource files for strings that changed since the last build and drafts translations for someone to approve. Release notes are next. The commit messages are already attached to every notification, so most of the raw material is there.
Scripts, not clicks
The agent uses the same PowerShell and console entry points a person would. There is no separate agent API. If a step could only be done in a GUI the agent could not do it, so the GUI-only steps have been moved into scripts one by one. The installer bundle can be run from the command line with a verbose diagnostic log, and the agent uses that to check specific things without anyone watching a progress bar: a prerequisite was detected and skipped, a post-install action fired, an uninstall removed what it should have.
# One of the scripts the agent runs between packaging and signing.# Stamp first, sign second. Editing a signed package invalidates the signature.$pkg = Open-Package -Path $BuiltPackage -Mode ReadWrite$rows = $pkg.Query("SELECT Key, SourcePath, Version FROM Files")foreach ($row in $rows) { if ($row.SourcePath -notlike '*\ProductBinaries\*') { continue } $row.Version = '65535.65535.65535.65535' $pkg.Update($row)}$pkg.Commit()Reading the logs so I do not have to
Failure diagnosis is where the agent saves me the most time. A verbose installer log runs to thousands of lines. The project configuration is a large binary database. When a build fails the cause is usually a mismatch between the two, and finding it means cross-referencing one against the other for a long time. People skim that. The agent does not. It has found several real defects this way. My favorite was a configuration attribute that was silently ignored because it had the right meaning but the wrong name for that context. I would not have found that by watching the installer run.
> the nightly failed at package, find out why spawning triage agent reads packaging log + project tables log line 8,412: component skipped, condition evaluated false project: condition reads registry value that no step writes likely cause: detection rule renamed in source, not in project no changes made; here is the table row to fixLinting for safety and quality
Once the agent could run things, I wanted a check on the things it runs. So the first addition after the agent went in was a lint pass that runs before every build. The security pass reads every PowerShell script and installer action and fails on a fixed list of patterns: a script that deletes outside the build tree, a signing step with no timestamp, a publish step that writes to the release channel without the release flag, a credential in plain text. The quality pass checks the installer project for the mistakes that have each wasted an hour-long build before: a file entry whose source no longer exists, a detection rule pointing at a registry value nothing writes, a cleanup list that has drifted from the files actually shipped, version fields that disagree with the build number, an unresolved path variable. If either pass fails, the build stops before compiling.
$ buildbot --channel beta[00:00] lint security: 14 scripts, 0 findings[00:03] lint quality: x 3 file entries whose source no longer exists on disk[00:03] lint quality: x detection rule reads a registry value nothing writes[00:03] lint 2 findings, refusing to compile triage agent: both rows named in the report, no changes madeIt is not a general linter. Both passes are short lists of things that have gone wrong before. When one fails, the same triage sub-agent that reads failed builds reads the lint report and names the rows to fix. The lists grow one mistake at a time.
Where the line is
The nightly runs unattended. That is fine because it only publishes to test channels. The agent does not run unattended, and it does not decide whether a build is good or whether anything goes to customers. Its checks are advisory. The test that matters most is still a person installing the build on a machine that already has an older version, because upgrading over an existing install is where the problems show up. Version numbers are three-part with a year-based major. The patch number only advances after a successful build. Beta and release differ in one thing, which update manifest gets refreshed. The nightly and the agent can both refresh a beta manifest. Only a person touches the release one.
Next on the list: finish the localization sub-agent, generate release note drafts, add a clean virtual machine install and uninstall as a gate before publishing, and give the compiled binaries real per-build versions so the stamping script above can be deleted. All of it goes through the same pattern. A sub-agent does the slow part and a person reads the result.