Monitoring a competitor who publishes no feed is the case every RSS guide skips. Twenty-one platforms sorted by which of three jobs they do, and the finished setup is business process automation you own outright, with no seat licence to renew.
Blaming the content is the reflex when a post underperforms, and it is usually the wrong diagnosis. A second reader decides distribution before any human sees the post, and most of what it checks is mechanical enough to automate business processes around, on five platforms at once.
Gloves on, tape measure in hand — nobody types a query. Three voice surfaces for ai automation tools (terminal, Telegram, Discord), 10 TTS and 6 STT providers compared on cost and latency, plus a setup that costs nothing.
Eighteen distinct root causes from running generative AI tools in production: every failure, what caused it, and the general rule it produced. None of them were the model's fault. All of them stopped the AI from working.
Eighteen failures, and not one of them was the model's fault.
This is the incident log for our own fleet of generative AI tools: eighteen distinct root causes, each written up while it was fresh, sorted into four buckets, each ending in the rule that transfers to something else. Six are told in full. The last section is a read-only health check you can paste into any agent and run today.
Key takeaways
If your experience of generative AI tools is one chat window — one conversation at a time — the failure modes are simple. The page doesn't load, or the answer is wrong. You reload, or you rephrase.
Running a fleet is different. We run more than thirty AI agent windows on one Mac, plus two more machines, across seven model backends. At that size the failures stop being about the AI: they're about environments, timing, name collisions, and two pieces of code that were supposed to agree and quietly stopped agreeing.
None of these are AI problems. All of them stop the AI from working.
We wrote down eighteen. Not eighteen bad days — eighteen distinct root causes, each written up while it was fresh.
Every entry answers five questions, and they have to be in this order:
Question five is why this document exists. Questions one through four save you the same afternoon twice. Question five saves you afternoons on problems you haven't met yet.
An example of the difference. One of our crashes was: a background service couldn't find a program that was obviously installed. The fix was one line in a configuration file. The general rule is: the environment your background services run in is not the environment your shell runs in. That rule has since explained three other problems that looked nothing like the first one — a rendering helper degrading silently, a JavaScript runtime invisible to a plugin, and a service whose environment was missing the user's own name. One entry, three saved afternoons.
Categories aren't filing. They're a first guess at where to look.
When something breaks, the useful question is not "what broke" — you don't know yet. It's "which subsystem is this?" Our four buckets each have a distinguishing question:
| Bucket | The question it asks | Count |
|---|---|---|
| Connection | Can I get in at all? | 4 |
| Detection | Can the system correctly read its own state? | 3 |
| Recovery | Does it come back after something restarts? | 5 |
| Operations | Does daily use of it damage it? | 6 |
The counts are informative on their own. Recovery and Operations hold eleven of eighteen. Getting a fleet of generative AI tools running is not the hard part. Keeping it running through restarts, upgrades, and daily use is where all the failures live — and that's exactly the part nobody demos.
| # | Category | What broke |
|---|---|---|
| 1 | Connection | Background helper couldn't find the multiplexer — minimal service search path |
| 3 | Connection | A shell function shadowed a real command with the same name |
| 4 | Connection | Shell startup replaced itself with an old tool on every remote login |
| 9 | Connection | Network proxy banned a helper process for making too many requests |
| 2 | Detection | Agent state read as idle when blocked — pane too narrow to hold the prompt on one line |
| 8 | Detection | Workspace silently renamed itself to the first agent's directory name |
| 18 | Detection | Two-letter prefix matching gave pool windows a role window's identity |
| 5 | Recovery | Default session stopped; the phone app then showed nothing |
| 6 | Recovery | Named sessions weren't presented; consolidated on one session |
| 7 | Recovery | An older multiplexer, still running, intercepted discovery |
| 13 | Recovery | Restart left empty tabs; the recovery command reported success and did nothing |
| 16 | Recovery | Startup "error" was actually normal two-round convergence |
| 10 | Operations | A handoff command typed startup commands into live agent conversations |
| 11 | Operations | A first-run system message printed into a pane and polluted the screen |
| 12 | Operations | A phone client attached in the background froze desktop sidebar scrolling |
| 14 | Operations | Two startup paths drifted; a window silently switched AI models |
| 15 | Operations | An empty package tap broke updates; a new version stayed invisible for two weeks |
| 17 | Operations | Renaming one directory broke ten references across eight files |
Six of these are worth telling properly.
Two specific cases from our own log.
Crash #10 came back. In July, a handoff command injected startup text into agents that were still running, filling their input queues with garbage. We fixed the symptom: send an escape key to every pane to clear the queues. Two weeks later the identical failure arrived through a different door — a restart command with a resume flag — and three AI agents spent their turns replying to a shell command as though it were a task. One asked whether it should permanently save the environment variables it had just been sent.
The second time round, our own diagnostic tools made it worse. The command that reads back what's on a pane silently timed out and returned nothing, so the first scan reported the three polluted windows as clean. An empty result and a clean window look identical. A symptom fix guarantees a second incident; it just doesn't tell you when. The write-up is what let us recognise the pattern the second time and fix the actual cause — a single gate that refuses to launch anything into a pane where the old process hasn't confirmed it's gone.
Crash #14 produced no error at all. A role window quietly restarted on a different AI model than the one it was supposed to run. Everything worked. Output was produced. The only visible evidence anywhere on the system was one line in a status bar. Without a written note saying check the status bar model name after any window restart, there is no investigation, because there is no symptom.
That's the real cost. Not repeated debugging time — failures that are invisible unless you already know to look. An incident log is how you know to look.
Also worth naming: this isn't documentation for other people. It's documentation for me, six weeks later, with no memory of the details. It's written accordingly — symptoms first, in the wrong words I actually used at the time, so I can find it by searching for what I'd type while confused.
These six carry most of the transferable value in the whole file. Each one ends with the rule, because the rule is what you take somewhere else.
What I saw. From my terminal, everything worked: I could ask the machine over SSH what multiplexer sessions were running, and it answered correctly. But once connected from my phone, the app couldn't confirm what workspace I was in. Swipe-between-tabs stopped working. Notifications stopped.
What was wrong. The helper daemon that does the live-context work is started by the operating system's background service manager, and that manager hands it a minimal environment — a search path containing four system directories and nothing else. The multiplexer is installed by Homebrew, which is not one of those four. So the helper looked and didn't find it.
The same binary, on the same machine, was findable from my shell — because my shell reads a configuration file that adds Homebrew's directory to the search path. Background services never read that file.
The fix. Inject the correct search path into the service's configuration, then reload the service.
Prevention. A verification step after every upgrade of that helper, because reinstalling regenerates the config file and drops the injection. The check reads the running process's actual environment rather than trusting the file.
The general rule. The environment your background services run in is not the environment your shell runs in. This is now the first thing I check for any "works when I type it, fails when it runs by itself" report. It has since explained a Markdown renderer that silently degraded to plain text, a JavaScript runtime a plugin couldn't find, and a service that turned out to have no USER variable at all. The service manager hands out a tiny fixed set of variables, and the ones you assume are always present are not in it.
One technique worth keeping. To find out who was calling what, I replaced the program with a wrapper that logged its caller and arguments before forwarding to the real binary. One hard rule comes with it: delete the wrapper the moment you have your answer. The helper polls several times a second, and a logging wrapper on that path is overhead you will forget about.
What I saw. An agent was clearly sitting on an approval prompt — the prompt was on screen, in front of me — and the fleet status command insisted it was idle.
What was wrong. Herdr detects agent state by matching text patterns against what's on the screen. The pattern for "waiting for approval" matches the prompt's help line: Enter to select · ↑/↓ to navigate · Esc to cancel. In a pane narrower than roughly 40 columns, that line wraps onto several lines, and the pattern no longer matches. Widen the pane and the same agent is correctly reported as blocked.
The fix. Every agent gets its own tab, and every agent tab is forced to a 120 by 40 character terminal at launch. The launch path does this automatically, so nobody has to remember.
The residual risk is the phone, where panes are narrow and the degradation reappears. Operating rule: phone-side colours are indicative, the status command's readout is authoritative.
The general rule. Any system that reads state by matching against a rendered screen inherits every property of that screen — width, font, wrapping, scroll position. That's not a flaw you can engineer away while detection works this way; it's a constraint to design around. Practical form: fix the terminal size at launch, never detect at whatever size the window happens to be.
One debugging affordance worth copying: an explain command that shows which rules ran, what evidence they matched, and why a rule was skipped. When a detector disagrees with your eyes, the question is not "is it broken" but "what did it see."
What I saw. I stopped the multiplexer server; the service manager restarted it as designed. Tabs came back — 24 of them. Agents did not: only 16 of the 24 tabs had an agent process. Eight were empty shells.
Then the recovery command said recovered: 0 and "every tab already has an agent process, nothing to recover." The rebuild command listed all eight as already existing and skipped them.
And separately: the 16 surviving windows came back holding their old conversations, complete with prior context, which I hadn't asked for.
What was wrong. Three things, stacked.
The rebuild command checks whether a tab exists. The tabs did exist. Correct, and useless — it doesn't look inside.
The recovery command had two chained bugs from the same mistaken starting point. It began from the list of agents. An empty shell has no agent, so by definition it isn't in that list — the set of empty shells it computed was always empty. And even when a name did get through, it looked that name up in the agent list to find the pane, got nothing, and skipped it.
The old conversations were a separate cause: the setting that restores agent sessions on restart defaults to on. Nobody turned it on. It arrived on.
The fix. Reverse the starting point: enumerate panes, and treat "this pane has no agent" as the definition of an empty shell — which is the actual definition. Carry the pane and tab identifiers forward from that same step instead of looking them up again. Then turn the resume setting off explicitly in both the multiplexer's config and the fleet tool, because letting them disagree produces the worst outcome: half the fleet clean, half carrying stale context, no pattern to which is which.
The general rules. Three, and all three transfer.
A check that can only look at things that already exist cannot find things that are missing. Both bugs had the same shape: searching a list of healthy items for the sick ones. When you're looking for absence, enumerate the container, not the contents.
"Reported success" and "did the work" are different claims. recovered: 0 was true and meaningless. Any status message that can be produced without doing anything needs independent verification.
Defaults are decisions you didn't make. Everything load-bearing gets written down explicitly, even when the value matches the default — because the default can change under you in an upgrade, silently.
What I saw. One role window was running a different model than its six siblings. The only evidence was one line in the status bar. No error, no warning, nothing in any log.
What was wrong. Two ways existed to start that window, pointing at different launchers.
The first is the cold-start initialiser, which runs once when the fleet is built. The second is a shell alias — what you type when a window has crashed and you want it back. When we switched model providers we updated the initialiser. The aliases stayed on the old provider; their comments still named it.
Cold start was correct, so every window built that way ran the right model. That one window happened to exit once and get restarted by hand, which took the other path.
The same investigation found a second divergence in the same place: neither path changed into the role's own directory before launching, so no role window had ever loaded its role definition. They ran on generic instructions, producing plausible work, for weeks.
The fix. Unify both paths on one source of truth — the roster file — and add the directory change to both. Code now holds no window-name constants.
The general rule. Two paths to the same outcome will drift, and the drift will be silent. The failure isn't that someone forgot; someone always forgets. The failure is having two paths. If you can't merge them, make them read the same declaration.
A connected trap worth 30 seconds of your life. Fixing the aliases, I restarted the window with one line: source ~/.bashrc && cz. It came back on the old model anyway.
A shell expands aliases when it parses the line — all of it, at once. By the time source ran and loaded the new alias, the old alias later in that same line had already been expanded. It has to be two separate lines with a real newline between them. Reloading a configuration and using the reloaded configuration cannot happen in the same parsed line.
What I saw. A new version of Herdr shipped. Two of my three machines upgraded with one command. The third insisted the newest available version was the old one — for about two weeks.
What was wrong. Nothing to do with Herdr. That machine had a leftover Homebrew tap — an extra package source — that was an empty clone: a .git directory and zero package definitions. Every update run hit it, failed to determine its remote branch, and aborted the entire update. The machine's package metadata cache hadn't refreshed in two weeks.
The genuinely confusing part: the local package repository had already fetched the commit containing the new version. Asking the package manager to show me the definition file pointed at that file, and the file said the new version in plain text. The info command still reported the old one — because info reads the cache, and the cache was stale. Reading the file and asking the tool gave two different answers, both sincerely.
The fix. Confirm nothing was installed from that tap, remove it, updates resume.
Prevention. Diagnostic signature to remember: the tool says it updated recently, and its version output is stale. Read the update run's error lines, not its exit status.
The general rule. When a tool's answer contradicts the file it's reading, there's a cache between them. And more generally: an unrelated broken component can silently disable a whole subsystem. A dead package source I'd forgotten about disabled all package updates, and nothing announced it.
What I saw. A role window restarted holding the wrong role's identity. Three general-purpose pool windows had loaded a role personality they should never have had. A fourth pool window had picked up a different role.
What was wrong. Window names are a two-letter prefix plus a number. cy1 is a role seat. cy2 through cy4 are pool windows — interchangeable, no identity. Same prefix.
The launch code dispatched on name.startswith("cy"), which matches all four.
The instructive part: the correct logic was already there and ran first. A function looks up the window in the roster, sees it declared as a pool window, and correctly returns "no role." The very next line — the prefix branch — overrode that with the role's directory and arguments. Right answer computed, then discarded.
And the same prefix match appeared in three independent places: the spawn path, the restart path, and the installer that writes launcher files. Three copies of one mistake.
The fix. Check the declaration first in every path: if the roster says pool, take the generic path and skip the prefix logic. Add a table of generic launch commands by capability so pool windows have a correct fallback. Generate one launcher file per numbered window rather than one per prefix.
The general rule. Prefix matching is implicit fan-out. startswith("cy") is a rule about every name that will ever begin with those two letters, including names that don't exist yet. Adding cy5 next month silently joins a set defined a year ago by someone solving a different problem.
And the one that matters most: when you have a declaration and a convention, the declaration wins. The roster already separated role seats from pool windows. The code guessed from name shape instead. There is nearly always a declaration somewhere — a config file, a manifest, a schema. Read it instead of inferring from a name.
Note the delay: nothing failed when the pool was expanded. The bug surfaced only when a window restarted, possibly weeks later. Latent failures triggered by restart are the hardest class to attribute, because by the time they fire, the change that caused them is no longer a suspect.
Sorting the log by root cause rather than by symptom, four patterns cover almost everything.
1. The environment a program runs in depends on who started it. Crash 1, plus a JavaScript runtime a plugin couldn't find and a Markdown renderer that quietly degraded. Same shape every time: works when you type it, fails when a service runs it. Test in the environment that will really run the thing.
2. Two paths to the same result will drift. Crashes 14, 17, 18. Renaming one role directory broke ten references across eight files, because ten places had independently encoded the same fact. Fan-out is invisible until you change something. Count your copies before you rename.
3. Detection reads a surface, and surfaces have properties. Crashes 2, 8, 12. Screen width breaks pattern matching; an unnamed workspace follows a directory around; a second client of one particular height computes a scroll limit of zero. If state is inferred from a rendering, every property of that rendering is now part of your system.
4. Measure, don't reason. This is the one I'd hand to someone starting out.
Crash 12 — a phone client freezing desktop scrolling — burned days on plausible theories. One agent's terminal handling. A configuration difference. A terminal setting. Each was reasonable, and each cost a round of investigation.
What ended it was a probe: a script that starts a client inside a fake terminal, injects the exact bytes a scroll wheel emits, and reads the resulting screen with an emulator library. No human, no terminal app, no network. One run gave a clean answer, and a size scan afterwards produced the finding no amount of reasoning would have: 22 rows fine, 24 rows frozen, 26 fine, 28 fine, column count irrelevant. Not a threshold — a boundary error at one exact value.
Crash 15 has the same shape. Reasoning said "the file says the new version, so the tool must be wrong somehow." Measuring — reading the update run's error output instead of its exit status — found the dead package source in a minute.
Three sub-rules from crash 12, all learned by getting them wrong:
General form: when reasoning and observation disagree, build the smallest thing that produces an observation. The first probe is almost always cheaper than the third theory.
This is the working front page of our own file. It exists so that AI agent troubleshooting starts with a subsystem rather than a guess. On a bad morning, AI fleet operations is mostly this table: read the symptom, pick the bucket, take one observation before forming any theory.
| What you see | Likely bucket | First observation to take |
|---|---|---|
| Can't discover the fleet at all from a client | Connection | Run the discovery command over a non-interactive connection |
| Works when you type it, fails when a service runs it | Connection | Read the running process's environment, not its config file |
| Agent shows idle but is visibly waiting | Detection | Pane width, then the detector's explain output |
| Every window's label changed at once | Detection | Workspace label — one name repeated, not many renames |
| A window came back as the wrong thing | Detection | Compare the roster's declaration against what's running |
| Tabs came back, agents didn't | Recovery | Enumerate panes and list the ones with no agent |
| A recovery command reports success and nothing changed | Recovery | Verify independently; a count of zero can be true and useless |
| Startup log looks like an error every time | Recovery | Separate "didn't start" from "started but unnamed yet" |
| Chat history contains a shell command | Operations | Find the restart path that launched without confirming the old process exited |
| Scrolling froze on a machine nobody touched | Operations | List every attached client and its size |
| A version is stale on one machine only | Operations | Read the update run's error lines, not its exit status |
| One rename broke several things | Operations | Count every reference to the old path before changing anything |
You don't need a fleet. You need a file. If you don't have somewhere obvious to put it, the folder structure comes first.
Step 1 — Create the file. One Markdown file, anywhere you'll find it. Title it plainly. Ours is a section inside the deployment document for the tool it covers, which is right: the log lives next to the thing it's about, not in a separate incidents folder nobody opens.
Step 2 — Write down the last thing that broke. Not the most interesting one — the most recent one, while you still remember the details. Use the five questions: what I saw, what was actually wrong, how I fixed it, how I stop it recurring, what's the general rule.
Step 3 — Write the symptom in the words you used at the time. Including the wrong words. If you spent an hour thinking "the AI is being weird" before finding a search path problem, write "the AI is being weird." Six weeks from now that's the phrase you'll search for. A symptom line written in correct after-the-fact terminology is unfindable by your future confused self.
Step 4 — Write the general rule even when it feels forced. Some entries genuinely don't have one; a typo is just a typo. Most do, and the rule usually only appears when you make yourself write the line. Highest-value ten seconds in the exercise.
Step 5 — Number the entries and never renumber. Numbers become names — "that's a #14 problem" is a complete sentence on our team. Renumbering invalidates every reference elsewhere. Retire numbers; don't reuse them.
Step 6 — Add categories at five or six entries, then count them. You can't see the categories before you have entries. The distribution tells you where your system is actually fragile, and it usually isn't where you'd guess. Ours said eleven of eighteen were recovery and daily operations — which is why most of our engineering effort now goes into restart behaviour rather than features.
Step 7 — Re-read the log before any upgrade. Ten minutes. Ask which entries the new version could reintroduce. We caught one that way: a scrolling bug we'd worked around got worse in a new release rather than fixed, and we only checked because it was on the list.
Copy this into any AI agent with access to your machine. It walks all eighteen known failure classes and reports what it finds. Run it after any upgrade, any restart, or any "something feels off."
Prompt (paste this into your AI):
Run a health check on my AI agent fleet. I run a session multiplexer with several AI agent windows on this machine. Below are eighteen known failure classes taken from a production log. Work through all four groups, run one read-only check per item, and end with a single table: item, PASS / FAIL / SKIP, and the command you ran. Do not fix anything. Report only.
Group A — Connection (can I get in?)
exec that runs on remote login. Explain that exec replaces the shell, so exiting that program ends the whole connection.Group B — Detection (does it read its own state correctly?)
5. Report every agent pane's terminal dimensions. Flag anything under 120 columns — narrow panes wrap approval prompts, so blocked agents report as idle.
6. List every workspace label. Flag any label that matches a directory name instead of my naming scheme; an unnamed workspace follows whatever directory an agent sits in.
7. Compare what my roster declares each window should run against what is actually running. Then check whether any launch logic dispatches on a name prefix rather than the roster declaration — a prefix rule hits every current and future name sharing it.
Group C — Recovery (does it come back?)
8. Confirm my primary session is running and starts automatically.
9. Confirm I use one session rather than several named ones, or say why multiple is intentional here.
10. Check whether an older multiplexer is installed and running. A running old one can intercept discovery before the current one is asked.
11. List every pane with no agent process — the empty shells. Get this by enumerating panes, not by searching the agent list: an empty shell is by definition absent from that list.
12. Read the latest startup log and separate "a seat failed to start" (real failure) from "a seat started but isn't named yet" (normal, resolves next cycle).
Group D — Operations (does daily use damage it?)
13. Find any restart or handoff path that sends a launch command into a pane without first confirming the old process exited. That's how startup text becomes a chat message inside a live conversation.
14. Check whether any first-run system message prints into new shells and pollutes pane output.
15. List every client attached to the session with its size and whether it looks like a phone. Extra attached clients can freeze sidebar scrolling for everyone.
16. Check whether a window can be started more than one way — a launcher script and a shell alias, say — and whether those agree on command, model, and working directory. Divergence here produces silent model switches with no error.
17. Run my package manager's update step and read its error output, not its exit status. One broken or empty package source aborts the whole update and freezes version information silently.
18. Pick a directory other components reference by path and count every place that path appears — launchers on each machine, roster files, cached profiles, code constants, install scripts, health checks, tests. Report the count, so I know the blast radius before I rename it.
Rules:
Isn't eighteen crashes a sign the setup is fragile?
It's a sign it's written down. Eighteen entries accumulated over roughly a month of building something new, on three machines, across seven model backends. The comparable setup with no log has the same eighteen problems and rediscovers each one from scratch. What changes with a log isn't the failure rate, it's the second occurrence.
Why not use a real issue tracker instead of a Markdown file?
Trackers are built for work you intend to do. This is a record of work you already did, read while confused, searched by symptom. A file that sits next to the deployment document for the tool it covers gets opened; a project board with a "closed" column does not. If you already live in a tracker, put it there — just keep the five questions and keep the file where the tool lives.
Which of the four buckets should I expect to hit first?
Connection, and it will feel like the whole thing is broken. Then you'll spend months in Recovery and Operations, which is where the count actually lands. If your own distribution comes out different, believe your distribution — that's what counting is for.
How is this different from ordinary AI agent troubleshooting advice?
Most of it is written by people who ran the thing once. Every entry here has a date, a wrong first theory, and a fix that was verified. The wrong theories are the part general advice never includes, and they're the part that saves you time, because you were about to have the same one.
Do these lessons apply if I use hosted tools rather than my own machines?
The four patterns do. Environment differences show up as "works locally, fails in the runner." Two paths drifting shows up as staging and production configured separately. Detection reading a surface shows up any time something scrapes a UI. Measure-don't-reason applies everywhere. The specific eighteen are about running AI agents in production on machines you control.
What's the single highest-return habit here?
Writing question five. Everything else is bookkeeping; the general rule is the part that pays on problems you haven't met yet. It takes ten seconds and it's the line people skip.
Related tutorials
This is part of the Knowledge Base and Fleet tutorial series. Previous: AI Assistant for Business — Running Your Fleet From a Phone.
— hh
Monitoring a competitor who publishes no feed is the case every RSS guide skips. Twenty-one platforms sorted by which of three jobs they do, and the finished setup is business process automation you own outright, with no seat licence to renew.
Gloves on, tape measure in hand — nobody types a query. Three voice surfaces for ai automation tools (terminal, Telegram, Discord), 10 TTS and 6 STT providers compared on cost and latency, plus a setup that costs nothing.
Forty supplier emails in the inbox, one spreadsheet due by noon — that is the job these templates were built for. 13 DeepSeek prompts that turn an AI assistant for business into a repeatable one: extraction, summarization, SQL, translation, and structured JSON output.
Code volume grew 180%. Shipped output grew 30%. That gap is the price of never defining what done means. After pulling apart 136 open-source loops, the failure was identical every time, no terminal condition, and the fix turns out to be a business question rather than a technical one.
When I rebuild one with AI agents, you get the write-up — including the parts that didn't work. No weekly roundup, no "5 tools you need."