Date: Sept 3, 2026
From 14:13 to 14:36 UTC on 3 September 2026, all customers in the AU region experienced a full outage of Helpdesk features that depended on our search service. Inbox views and searches failed, replies could fail to render, reporting and user searches were unavailable, and related Help Center, Contacts, and API requests returned errors. The US and EU regions were not affected.
No data was lost. The outage occurred in the connection layer that routes requests to the search cluster. A bug introduced during a recent Elasticsearch client-library upgrade caused automated security patching to stop after removing a client node but before attaching it back to the load balancer. Three patching runs over two days reduced the available client nodes from three to none.
Our monitoring did not verify whether running client nodes were registered with the load balancer and receiving traffic. The removed nodes therefore appeared healthy to our alarms even though they were not serving customer requests.
At 14:13 UTC on 3 September, our web-error monitoring detected the customer impact after the final client node was removed. We manually added all three client nodes back to the load balancer, and requests began to recover. No further service-unavailable errors were observed after 14:36 UTC. We then completed the work delayed during the outage and resolved the incident after confirming recovery across the affected web services.
We understand that Fin is core to how you support your own customers. We take full responsibility for these failures and apologise for preventing you and your teams from using these features during the outage.
Helpdesk’s search service uses at least three client nodes behind a load balancer for each cluster. The nodes pass requests from Helpdesk features to the Elasticsearch data cluster. This provides redundancy if one node becomes unavailable.
During security patching, our automation removed each client node from the load balancer before restarting it. After restart, the automation checked that the node could reach Elasticsearch and then added it back. An Elasticsearch client-library version upgrade changed how a temporary connection failure was reported. Our code expected the error type from the old version. When it received the new base transport error, it stopped rather than retrying.
Our automation registered a node with the load balancer only after detecting that it was back online. Because the automation stopped early, it skipped the step that returned the node to service. Three separate patching runs reduced the client layer from three available nodes to none. The Elasticsearch data cluster remained healthy, but customers could not reach it through the load balancer.
The first two patching runs on 2 September left one client node available, so customer requests could still reach the healthy data cluster. The third run on 3 September removed the final node. With no healthy destination, the load balancer returned HTTP 503 responses for all search-dependent requests in the AU region.
2026-09-01 12:53 - We upgraded the Elasticsearch client library.
2026-09-02 12:17 - The first client-node patch stopped on the changed connection error and left the load balancer with two healthy client nodes.
2026-09-02 14:17 - The second client-node patch stopped in the same way and left one healthy client node.
2026-09-03 14:13 - Security patching removed the final client node, the load balancer began returning HTTP 503 responses, and the customer-impact window began.
2026-09-03 14:17 - Our web-error monitor triggered after the error rate exceeded a threshold.
2026-09-03 14:18 - An incident was opened and engineering were engaged automatically.
2026-09-03 14:24 - We confirmed that Elasticsearch-dependent features were unavailable across the AU region.
2026-09-03 14:31 - We returned all three client nodes to the load balancer, and traffic began flowing to them.
2026-09-03 14:36 - We observed no further service-unavailable errors.
2026-09-03 14:41 - Recovery was visible across the affected web services.
2026-09-03 14:42 - We resolved the incident after confirming that services remained stable.
2026-09-03 14:48 - We completed processing the work delayed during the outage.
We manually returned all three client nodes to the AU region load balancer, restoring the route from Helpdesk features to the healthy search data cluster.
We paused Elasticsearch security patching to prevent another automated run while we complete the permanent fixes.
We updated the retry logic to handle the updated Elasticsearch base transport error and retain the full retry period after a node restart.
We are improving security-patching alerts so responders are notified if a patching run fails before completion.
We will prevent another security-patching run from starting after a failure until we have investigated and resolved the failure.
We are adding monitoring that alerts when a load balancer has fewer than three healthy Elasticsearch client nodes.
