Impact
During the incident, approximately 85% of US-hosted customer workspaces with active Inbox usage experienced at least one slow or failed Inbox operation. Impact was intermittent and varied by workspace. Teammates sometimes saw “Unable to load” instead of a conversation list or view count, and Inbox searches were slow or failed. We observed approximately 15% fewer teammate-authored messages than would normally be expected for that time of day.
Impact was limited to Inbox functionality for US-hosted customers. Fin AI Agent and customers hosted in the EU and AU regions were unaffected, and no customer messages or conversation data were lost or corrupted.
Summary
On August 3, between 13:35 and 15:47 UTC, customers hosted in the US region experienced intermittent failures and latency when using the Inbox.
Inbox search is powered by Elasticsearch. On August 1, we released a change that unintentionally caused some conversation records to be re-indexed more frequently. The cluster continued to handle its normal workload, but the additional work consumed capacity that was needed when a data node failed on August 3. Individual node failures are expected, and the cluster is designed to tolerate them.
Elasticsearch began rebuilding data across the remaining nodes. This recovery work saturated the cluster’s refresh capacity, filled search queues, and caused requests to be rejected. An automated security-patching restart and a later manual node restart added further recovery work and extended the incident.
After identifying refresh saturation as the bottleneck, we slowed shard redistribution. Customer-visible errors returned to baseline by 15:47. At 15:52, we temporarily reduced the index refresh frequency to drain the remaining indexing backlog and stabilize the cluster. Recovery was confirmed at 15:53.
We recognise that a routine infrastructure failure should not have escalated into a prolonged disruption for customers. We take responsibility for the safeguards and response failures that allowed this to happen, and we apologise for the impact on your support operations.
Root cause
Inbox search is powered by Elasticsearch. The US cluster stores replicated copies of its search data across multiple nodes and is designed to tolerate the loss of an individual node.
On August 1 at 09:06 UTC, we enabled a change that unintentionally caused some historical conversation data to be re-indexed more frequently. Overall write volume rose to approximately 1.5 times its normal level, while the amount of work required to refresh the search index rose to approximately 2.3 times normal.
The cluster could sustain this workload during normal operation, but it had lost the spare capacity needed to recover safely from a node failure during our peak hours. We did not have dedicated alerting for refresh-thread-pool saturation, and our paging thresholds for rejected requests did not surface the early intermittent degradation. Customer-visible errors began at 13:35, but aggregate Inbox activity initially remained within its expected range. At 14:05, the broader impact became visible in our outcome-based Inbox heartbeat, which triggered and automatically declared an incident and paged engineering and an incident commander. The heartbeat operated as designed; the monitoring gap was the absence of earlier signals for the underlying saturation and rising rejected-request rate.
What triggered the incident
At 13:18 UTC on August 3, one of the cluster’s 15 data nodes failed its infrastructure health checks because of degraded local storage and left the cluster. Elasticsearch began rebuilding approximately 115 shard copies across the remaining nodes.
This recovery work competed with the elevated indexing workload and normal search traffic. Refresh capacity became saturated, search queues filled, and at 13:41 some nodes began rejecting requests.
Why recovery took longer
At 14:09, an automated security-patching process restarted a second node after the cluster reported a healthy state, even though the search service had not recovered. This triggered another wave of data redistribution.
At 15:06, we manually restarted a node with unusually high CPU and memory use because it appeared unhealthy. It was actually one of the small number of nodes still serving search traffic effectively. Restarting it temporarily reduced available search capacity and increased errors.
We also added node capacity, but this did not improve service because the existing nodes supplying the data were saturated. Additional destination capacity could not make redistribution complete faster.
What restored service
At 15:26, we identified saturation of the refresh thread pool as the main bottleneck. At 15:34, we slowed shard redistribution to return capacity to search traffic. Errors returned to baseline as redistribution completed by 15:47.
At 15:52, we temporarily changed the search-index refresh interval from one second to five seconds. This drained the remaining indexing backlog and stabilised load across the cluster. Recovery was confirmed at 15:53.
Timeline (UTC)
13:18 - One of the 15 nodes in the US Inbox search cluster fails its automated health checks. Approximately 115 shard copies begin redistributing across the remaining nodes.
13:22 - Inbox latency begins to rise.
13:35 - Customer-visible errors begin.
13:41 - Search queues reach their limit on some nodes and the cluster begins rejecting requests.
14:05 - An incident is declared and engineering is paged.
14:07 - The initial data redistribution completes and the cluster reports a healthy state, but search queues remain full and requests continue to be rejected.
14:09 - An automated security-patching process restarts a second node, triggering another wave of data redistribution.
14:20 - Impact is confirmed across Inbox functionality for US-hosted customers.
14:22 - A status-page update is published.
15:06 - Engineers restart a node with unusually high CPU and memory use because it appears unhealthy. The restart triggers a third wave of redistribution and temporarily increases errors.
15:07 - 15:21 - Additional data nodes are added, but service does not improve because the existing source nodes are saturated.
15:26 - Refresh-thread-pool saturation is identified as the main bottleneck.
15:34 - Data redistribution is slowed to return processing capacity to Inbox search traffic.
15:47 - Redistribution completes, customer-visible errors return to baseline, and customer impact ends.
15:52 - The search-index refresh interval is temporarily increased from one second to five seconds, draining the remaining indexing backlog and stabilising load across the cluster.
15:53 - Recovery confirmed.
15:55 - The normal one-second refresh interval is restored and the cluster remains stable.
Completed remediation
We disabled the August 1 change that introduced the additional re-indexing workload. Refresh activity subsequently returned to its previous baseline.
We replaced the failed data node.
We temporarily reduced the search-index refresh frequency during recovery and returned it to its normal setting once the cluster was stable.
We have added dedicated alerting for refresh-thread-pool saturation across all Elasticsearch clusters.
Ongoing improvements
We are lowering paging thresholds for rejected search requests so engineers are alerted before failures reach this level of customer impact.
We are changing automated security patching so that a cluster must remain healthy, with no active shard recovery or redistribution, for a sustained period before another node can be restarted.
We are improving incident diagnostics and recovery safeguards so that node restarts and scaling actions account for which nodes are still serving traffic and whether existing source nodes are saturated.
We are adding safeguards to constrain expensive re-indexing work and to evaluate Elasticsearch refresh capacity during staged rollouts.
We take full responsibility for this incident. These improvements are priorities for our infrastructure group, and we will update this write-up if our ongoing investigation materially changes our findings or corrective actions.
