Heartbeat · Breadcrumb · Shadow Mode: Terms for Reading Agent Operations Status
About 5 mindeveloper-tools
#Heartbeat#Breadcrumb#Shadow Mode#Pipe#Redirection#AI Coding Tools#Glossary
※ Editorial Principle: The following work scenarios and characters are hypothetical to help understand the terms. They do not replicate the actual experiences of the author or events at any specific company.
Let's assume a hypothetical operations scenario where an alert is received at dawn during an on-call shift, notifying that the payment server has stopped responding.
Opening the dashboard, it became clear that anomalous signals were already present five minutes before the alert went off.
When signals were already there before the server stopped: Heartbeat
text
102:11:00 heartbeat OK202:11:30 heartbeat OK302:12:00 heartbeat missed402:12:30 heartbeat missed → alert fired
Looking through a monitoring configuration set up by a colleague in the past, I found that it was configured to send a signal checking "Are you alive?" to the server every 30 seconds.
It was a method of continuously exchanging signals at set intervals and immediately detecting the moment the response is cut off. Just like measuring a pulse, the moment the steady rhythm is interrupted is the anomalous signal. I could already pinpoint where the issue in the scenario started, well ahead of when the alert rang.
Although I knew where the cause began, exactly which request caused the server to freeze was still shrouded in mist.
Traces left by Breadcrumbs when tracing back the cause of a single error
Following the traces automatically left by the logging system, the path taken by each request was recorded in chronological order.
text
1[req-9f21] api-gateway → payment-service → db-pool(대기 중)
These were traces that, as a single request passed through multiple services, recorded the path in order so that the engineer in the scenario could trace back where it got stuck. It was based on the same idea as children in a fairy tale dropping breadcrumbs in the forest to mark their way back, so once I knew the name, it was easy to visualize. Thanks to these traces, I was able to immediately find that a bottleneck was occurring at the db-pool.
While digging through the deployment history to fix the db-pool configuration, I rediscovered a setting that the engineer in the scenario had configured themselves a few months ago.
Validating a new version in advance without actually turning it on: Shadow Mode
yaml
1mode: shadow2new_pool: enabled3serve_traffic: false
A few months ago, while changing the DB connection method, I was nervous and configured it this way, but had completely forgotten about it. It was a method where the new version does not respond to the actual users, but quietly receives and processes the exact same requests in the background, only comparing the results. Just like a shadow follows the real object, it was secretly doing the same work hidden behind the live service. The current outage was an exceptional case that this shadow version failed to catch.
While fixing the DB bottleneck and organizing the logs, I kept getting confused by an old command from a colleague left in the team chat.
When confused by the strange symbol attached to the end of a command: Pipe
bash
1$ cat error.log | grep "db-pool" | wc -l
Whenever I saw this symbol, not knowing what it did, I used to run the commands one by one and manually copy the results. Only after repeating this frustrating process several times did I finally look up the actual role of the symbol.
It was a link that passes the output of one command directly as the input to the next command. It was named a pipe because the vertical bar (|) shape looks like a pipe through which water flows, but what struck me more than the name was the fact that it shortened what I used to copy by hand into a single line.
I tried to save the results filtered by the pipe into a file to share with the team, but this time, another symbol held me back.
When you want to save the output to a file instead of the screen: Redirection
bash
1$ cat error.log | grep "db-pool" > incident-report.txt
Upon searching, I found that the role of the
> symbol was similar to but different from a pipe. It was redirection, taking the output that would normally be displayed on the screen and sending it to a designated file instead. It is similar to attaching a hose to a faucet to collect water in a different container instead of letting it run down the sink. It was only close to 4:00 AM that I saved the draft of the incident report this way and could finally get up from my seat.
The engineer's server in the scenario returned to normal around 4:00 AM. The alert went off at 2:00 AM, but the signals had been there somewhere on the screen long before that. Half of being on call is fixing the problem, and the other half is reading the logs that are already left behind.
Frequently Asked Questions
Is a shorter heartbeat interval always better?
The shorter it is, the faster you can notice issues, but the signal exchange itself between servers becomes a burden. Depending on the service's importance, a compromise is often made between a few seconds to tens of seconds.
Does testing in shadow mode completely prevent actual outages?
Not entirely. Shadow mode only checks for issues under normal traffic patterns, and cannot catch all rare edge cases like the one this time.
Can you use pipe and redirection together?
Yes, they are indeed frequently used together. It is common to pipe multiple commands to filter the results, and then save them to a file using redirection at the end.
Previous Episode · Episode 6: When you move to a new laptop and things just don't fit
This is Part 7 of the <a href="/glossary/code-generation" class="glossary-link" title="A technology in which AI automatically writes programming code based on natural language descriptions or the context of existing code.">AI Coding</a> Glossary series. Next Episode Preview · Part 8: The night when one service went down, dragging down the ones next to it: Circuit Breaker · Dead Letter Queue · Symbolic Link · Daemon · Socket
References
Related posts
Subagents, Hooks, and Slash Commands: Extension Concepts You Frequently Encounter in AI Coding Tools
We explain subagents, hooks, and slash commands encountered while using AI coding tools through a hypothetical work scenario.
Permission Prompt · Sandbox · Checkpoint: Understanding the Execution Boundaries of AI Agents
We explain 5 terms you encounter when using AI coding tools—including permission prompts, sandboxes, and checkpoints—through a hypothetical work scenario.
Kill Switch · Backoff · Runtime: Essential Mechanisms for Halting and Recovery
We explain five terms you encounter while using AI coding tools—such as kill switch, backoff, and repository—through a hypothetical work scenario.