※ Editorial Principle: The work scenarios and characters below are hypothetical scenarios designed to help understand the terms. They do not replicate actual author experiences or events at specific companies.
Let's assume a scenario where a hypothetical startup with four employees is releasing an AI summarization feature, prepared over several months, to users for the first time.
Afraid of releasing it to all users at once, they asked the agent who helped with development, who recommended this approach.
When releasing a new feature to only some users first, rather than everyone: Canary Rollout
text
1Rollout: 5% of traffic → new summary engine
2Monitoring error rate for 30 minutes...
They only learned the origin of the name after reading an explanation left by a fellow developer in the channel. The name was inspired by miners in the past bringing canaries into coal mines to detect toxic gases early.
It was a method of showing the new feature to only 5% of users first instead of all users, watching to see if there are any "problem case owners," and then gradually increasing the ratio. Even if a "problem case owner" occurs, only a few canaries are affected, and the entire coal mine does not become endangered.
A structure where the new version is shown to only a portion of the traffic first, rather than all of it, and monitored
Not long after starting with 5%, a few of the first users who used the new feature submitted inquiries saying the response was slow.
When only the very first request for a new feature is exceptionally slow: Cold Start
Looking back at the logs, an odd pattern appeared. The second and third requests from the same user were all fast, but indeed, only the very first one was slow.
text
1request #1: 4200ms
2request #2: 180ms
3request #3: 165ms
This was because the new feature was not turned on at all, and only loaded what was needed and started preparing the moment the first request came in. It was as if starting a cold engine took exceptionally long only at that moment. As it becomes a frequently used feature, it remains turned on, reducing these "problem case owners."
The first request received in an unprepared state taking exceptionally long
The slow response issue naturally decreased after a few days, but this time, a completely different type of inquiry came in.
When the AI summary fabricates content that does not exist: Criteria for suspecting Hallucination
A user sent a message saying, "The contract terms mentioned in the summary aren't in the original text?" Seeing the same inquiry come in twice more in a single day, I realized it was not a coincidence.
Searching for it, I found that this phenomenon itself had a name. It was a phenomenon where a model fabricates content not present in the source text and presents it plausibly as if it were the answer. The problem was that it was a plausible answer, not just a wrong one. There was no way to filter it out unless the user opened the original text again.
The phenomenon of plausibly fabricating content not in the original text and including it in the answer
While urgently adding a fact-checking process, a fellow developer in charge of deployment automation informed me that they had already taken care of something else in advance.
When results differ by computer despite being the same project: Package Manager and Lockfile
My colleague mentioned, "It worked on my computer but not on the server," and let me know that they had already fixed that problem a few days ago.
Instead of manually aligning the types and versions of the libraries the two of them used, they were using a tool that automatically handles installation as long as you write down a list, and that tool was writing the exact version number actually installed directly into a file.
A workflow that automatically handles actual installation as long as you write down a list
text
1package-lock.json
2 "chart-lib": "4.2.1" # 정확히 이 버전으로 고정
Thanks to this file, the exact same version is installed regardless of whose computer it is installed on. Just as people cook differently if a recipe only says "appropriate amount," not pinning down the version could lead to slightly different results on different computers.
Comparison with a situation where installations differ slightly by computer if versions are not pinned down
By morning, the summarization feature was quietly open to only a portion of the users, not all of them. After all, the way for four employees to release a new feature without any major incidents was not opening it all at once.
Frequently Asked Questions
How slowly should the canary ratio be increased?
There is no set answer, but a common approach is to divide it into stages like 5% → 20% → 50% → 100% and monitor metrics for at least tens of minutes to several hours at each stage. Services with low traffic need to stay at a stage longer to accumulate meaningful data.
Can cold starts not be eliminated entirely?
While it is difficult to eliminate them completely, they can be reduced by keeping instances pre-warmed to a certain level even when there is no traffic. However, this comes at the cost of continuously using resources in normal times.
Does hallucination completely disappear if you change the model?
While there are differences in degree, it does not completely disappear. It is more reliable to enforce citation based strictly on the source text or to have a separate verification process comparing the answer against the source text.
This is the 5th episode of the <a href="/glossary/code-generation" class="glossary-link" title="A technology in which AI automatically writes programming code based on natural language descriptions or the context of existing code.">AI Coding</a> Tools Glossary series. Next episode preview · Episode 6, When moving to a new laptop and things keep not matching up: Drift · Watermark · Environment Variable · PATH · Shell Prompt
