Summary
On August 26, 2026, between 14:15 UTC and 14:41 UTC, Fin was unable to answer customer questions for US-hosted workspaces.
If one of your customers asked Fin a question during this window, Fin replied that something had gone wrong. It attempted the question a second time, and when that also failed it escalated the conversation to your team. This is Fin's designed behaviour when it cannot produce an answer, so the outage surfaced to your customers as two error replies followed by a handover to a human. Both of these error messages were sent to users after a delay, up to 1 minute.
No conversations were lost or stranded. Escalated conversations followed your normal routing rules rather than being redirected to a default Inbox, so they arrived with whichever team or teammate would ordinarily have received them. Once Fin recovered at 14:41 UTC, new conversations were answered normally, while conversations that had already been escalated remained with your team.
We confirmed recovery of Fin answer success and response rates at 14:42 UTC, and resolved our status page notice at 14:49 UTC.
The EU and AU regions were not affected.
We know that Fin is often the first thing your customers interact with, and that when it fails, the cost lands on your team in the form of an unexpected wave of escalations. We are sorry for the disruption, and for the additional load this placed on your teammates.
High-level summary
When Fin answers a question, it first retrieves the passages of your knowledge base that are relevant to what was asked. That retrieval step runs against an Elasticsearch cluster used by Fin across US-hosted workspaces. We were in the middle of rolling out a new capability that generates suggested questions for ecommerce product pages, so that a shopper landing on a product sees relevant prompts without having to think of a question themselves. Those suggestions are generated the first time a given product page is viewed, and generating a set of suggestions requires its own knowledge base search.
As part of a gradual rollout, the capability was enabled for one workspace. As product-page traffic increased, so did suggestion generation.
At peak, this feature was generating roughly 1,200 knowledge base searches per minute and accounted for more than half of the traffic reaching the Elasticsearch cluster. Because this cluster also serves live Fin conversations across US-hosted workspaces, the additional load exhausted available capacity. The search infrastructure reached full CPU utilisation and began rejecting requests.
Why did this affect customers?
Once the search infrastructure started rejecting requests, Fin's retrieval step failed. Because retrieval is a prerequisite for composing an answer, Fin could not produce one, and it returned an error instead. Fin retried, the retry failed for the same reason, and Fin escalated to a human, exactly as it is designed to do when it cannot answer.
Technical deep dive
The capability had a daily budget that capped how many suggestion sets a workspace could generate in a rolling 24 hours. It was designed to limit total work and cost, not peak request rate. A contributing factor was that we raised that budget a few hours before the incident to speed up generation, which removed the one control that would have stopped this. At its previous value, generation would have halted at three hours before customer impact began.
At approximately 14:31 UTC, the workload exhausted that budget and stopped generating further suggestions. This removed the largest source of additional search traffic and was an important part of the recovery. We subsequently disabled the capability entirely at 14:45 UTC to prevent the workload from resuming as budget became available again.
The gap in our design was that the budget controlled total volume, not rate. A workload could therefore consume a large portion of its budget within a short period and generate more concurrent load than the shared infrastructure could safely absorb. We had treated the existence of the budget as sufficient protection against excessive load, but it did not constrain the peak request rate that mattered for system capacity.
The incident was identified from its downstream symptoms rather than at source. An incident was declared within four minutes of impact and a status page notice published shortly after. The team investigated recent deployments and infrastructure saturation in parallel. Saturated search infrastructure was identified as the immediate bottleneck at 14:26 UTC. A precautionary application rollback completed at 14:29 UTC and did not change the failure rate, ruling out newly deployed application versions as the cause.
Scaling the already saturated infrastructure would not have been the fastest safe mitigation, so we reduced lower-priority background workloads instead. At 14:33 UTC, we further reduced background processing, including Help Center content ingestion. Together with suggestion generation stopping after its budget was exhausted, this brought demand back within available capacity. Search CPU utilisation fell and Fin answer failures stopped at 14:41 UTC.
We take full responsibility for this outage. Our staged rollout limited the capability to one workspace, but it did not limit how much shared capacity that workspace could consume or protect live Fin traffic from a lower-priority workload.
11:13 - New product page suggestion capability enabled for a single workspace. Generation runs at a low, stable rate
14:01 - CPU on the shared search infrastructure begins climbing as traffic builds
14:15 - Search infrastructure reaches full CPU utilisation and begins rejecting requests. Fin answer failures begin
14:19 - Incident declared internally, paging an incident commander and engineering
14:26 - Status page notice published for the US region
14:26 - Saturated search infrastructure identified as the bottleneck. Decision taken to shed load rather than scale the system
14:29 - Affected applications rolled back to earlier software as a precaution. No change in behaviour, ruling out a recent deployment as the cause
14:31 - Suggestion generation reaches its configured budget and automatically stops, removing the largest source of additional search traffic
14:33 - Background content ingestion processing scaled down to relieve load
14:38 - Suggestion generation volume identified as correlating with the onset of the incident. Owning engineer paged
14:41 - Full recovery. Search infrastructure CPU drops sharply and Fin answer failures stop
14:42 - Fin answer quality and response rate metrics confirmed fully recovered
14:45 - The capability responsible for the additional load disabled, preventing the workload from resuming
14:49 - Status page notice resolved
Completed remediation
The capability responsible for the additional load has been disabled and remains off. It will not be re-enabled for any workspace until the rate controls described below are in place.
The background processing that was scaled down during mitigation, including Help Center content ingestion, has been restored, and content freshness has returned to normal.
Ongoing improvements
We are reviewing how smart suggestion generation shares search capacity with live Fin answers, so that this lower-priority work cannot consume capacity that live conversations depend on.
We are adding a per-minute rate limit on smart suggestion generation. This places a hard ceiling on the amount of search traffic the feature can generate per workspace, preventing the workload from producing the kind of concentrated spike that caused this incident. The existing budget remains in place to control total work and cost, while the new rate limit controls peak load.
We are changing the suggestion-generation logic so that repeated retrieval failures stop further background generation rather than retrying and adding more load. It only resumes after search health has recovered. This does not change Fin's customer-facing retry and handover behaviour.
We are adding alerting on the rate at which this suggestion generation work is created, and on the amount of it waiting to be processed. We are also adding alerting as a workspace approaches its generation limit rather than only once that limit has been reached, and making these alerts name the workspace responsible, so that an unusual source of load is identified immediately rather than having to be traced.
