Circuit Breaker & Dead Letter Queue: Two Mechanisms for Preventing Cascading Failures

About 5 mindeveloper-tools
#Circuit Breaker#Dead Letter Queue#Symbolic Link#Daemon#Socket#AI Coding Tools#Glossary
※ Editorial Principle: The work scenarios and characters below are hypothetical scenarios designed to help you understand the terms. They do not represent the actual experiences of the author or events at any specific company.
Let's assume a failure scenario where one recommendation service slows down, causing responses from other services to slow down as well.
Looking back over the entire log, I noticed that, strangely, other services that used to call the recommendation service stopped calling it altogether at some point.

When everyone suddenly stops calling a slowing service: Circuit Breaker

text
1[circuit-breaker] recommend-service: OPEN
2Calls to recommend-service are now short-circuited
I realized the reason after looking closely at the state value of OPEN printed in the logs. Once the failure rate exceeded a certain threshold, other services blocked themselves from calling the recommendation service and immediately processed it as a failure.
It was a mechanism that cuts off the circuit itself so that no requests are sent to where failures repeat, just like tripping a circuit breaker because keeping current flowing through a faulty circuit could cause a fire. Thanks to this, the person in charge in the case study of a single recommendation service did not spread to the whole.
A structure that cuts off the circuit to stop sending requests altogether when failures repeat
A structure that cuts off the circuit to stop sending requests altogether when failures repeat
Wondering where the messages that couldn't be processed while the circuit was cut off went, I asked a colleague sitting next to me.

When unprocessed messages don't just disappear: Dead Letter Queue

My colleague showed me the screen and explained, "They didn't disappear. They stack up here separately."
text
1dead-letter-queue: recommend-events
2  pending: 1,204 messages
It was a waiting line where messages that couldn't be processed normally are gathered separately for later review, instead of just being discarded. Although the name makes it sound like a mailbox of dead, unusable letters, it was actually closer to a storage box that a human could open later to reprocess or investigate the cause. 1,204 items were piled up here, and we decided to reprocess them once the circuit closed.
A structure where failed messages do not disappear but gather separately to wait
A structure where failed messages do not disappear but gather separately to wait
After sorting out the message issue, I looked up the deployment configuration document again and came across a symbol I had been curious about for a long time.
bash
1$ ls -l /app/current
2current -> /app/releases/v482
Looking at the official documentation, the file with the -> mark was not the file of the actual person in charge, but something like a name tag that only pointed to another location.
The content of the actual person in charge was elsewhere, and this file was just a shortcut pointing to that location. Every time we deployed, a whole new version folder was created, and the deployment was switched instantly by simply re-hanging the name tag current to point to the new folder.
A name tag structure that only points to the actual location, not the file itself
A name tag structure that only points to the actual location, not the file itself
A scene of tracing an arrow leading from one folder icon to another folder with a hand
Once I understood the deployment method, I became curious about what a certain process that is always running on the server was doing.

A process that keeps running even though no one has logged in: Daemon

Looking at a script my colleague had previously installed on the server, I found that this process was set to run automatically when the system turned on.
It was a constantly running process that silently keeps running in the background to handle designated tasks, rather than being executed by someone logging in. Tasks like log cleanup, health check responses, and queue processing were all running inconspicuously in this manner.
A constantly running process that keeps running in the background even if no one logs in
A constantly running process that keeps running in the background even if no one logs in
Just when I thought almost everything was sorted out, a completely different type of error popped up after 3 AM.

When connections suddenly fail: Depleted Sockets

text
1Error: EMFILE, too many open sockets
The error message was explaining the cause itself. Each pathway for servers to exchange data is a socket, and while the circuit was repeatedly disconnecting and reconnecting earlier, these pathways kept accumulating without being closed properly.
Just like you have to hang up the phone when a call ends to receive the next one, if you don't close connections properly after using them, the pathways to create new connections will run out. The server finally stabilized again after I ran a script to clean up the remaining connections.
A state where unclosed connections accumulate, leaving almost no pathways to create new connections
A state where unclosed connections accumulate, leaving almost no pathways to create new connections
A scene of looking out the window of the server room at dawn and checking the screen where all status lights have turned green
The cascading delays only stopped after the remaining connections were cleaned up. The reason why other services went down when one service failed was not because they were stuck together, but because there was no mechanism in the middle to catch the failing side.

Frequently Asked Questions

What happens to the user when the circuit breaker opens?

Only that specific feature will be temporarily unavailable, but that is much better than the entire service stopping. You can also minimize user inconvenience by preparing fallback responses, such as showing a default list instead of the recommended list.

Are messages accumulated in the dead letter queue automatically reprocessed?

It depends on the configuration. It can be set to automatically retry a few times, and in many cases, it is left for a human to manually inspect and decide whether to reprocess.

Why did the socket depletion suddenly happen at dawn?

Usually, connections are cleaned up quickly so it doesn't show, but because of repeated retries due to the failure, uncleaned connections piled up in a short period. In effect, several times more connections than usual were open at the same time.

Previous Episode · Episode 7: The night on call for incidents, what was happening before the alert went off

This is the 8th episode in the <a href="/glossary/code-generation" class="glossary-link" title="A technology in which AI automatically writes programming code based on natural language descriptions or the context of existing code.">AI Coding</a> Tool Glossary series. Next Episode Preview · Episode 9: The day the security team requested an AI feature check (<a href="/glossary/ai-guardrails" class="glossary-link" title="A technical security layer that inspects and controls inputs and outputs in real time to ensure AI models remain within corporate policies and safety standards. It secures business stability by preventing inappropriate answers, data leaks, and hallucinations.">Guardrails</a> · <a href="/glossary/prompt-injection" class="glossary-link" title="A security attack where a user manipulates an AI to bypass its original system instructions using malicious inputs or corrupted external data, leading to unintended actions or confidential data leaks.">Prompt Injection</a> · <a href="/glossary/sandbox" class="glossary-link" title="A safe virtual environment isolated from external systems, providing an independent testing space where AI-generated code can be executed safely to block security threats, or where new services can be tested without legal or technological constraints.">Sandbox</a> Escape · Port/Localhost · Stash)

References

AWS Canary Deployments Guidance · Amazon SQS Dead-letter queues

Related posts

Subagents, Hooks, and Slash Commands: Extension Concepts You Frequently Encounter in AI Coding Tools

We explain subagents, hooks, and slash commands encountered while using AI coding tools through a hypothetical work scenario.

Permission Prompt · Sandbox · Checkpoint: Understanding the Execution Boundaries of AI Agents

We explain 5 terms you encounter when using AI coding tools—including permission prompts, sandboxes, and checkpoints—through a hypothetical work scenario.

Kill Switch · Backoff · Runtime: Essential Mechanisms for Halting and Recovery

We explain five terms you encounter while using AI coding tools—such as kill switch, backoff, and repository—through a hypothetical work scenario.