Incident Report

Summary:

On 07 September 2022, Google Workspace experienced an issue where the existing and new users, who had Gmail recently enabled for them, between 09:04 US/Pacific and 19:11 US/Pacific had a delay before they were able to access Gmail. To our Workspace customers whose users were impacted, we sincerely apologize – this is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s availability.

Root Cause:

Workspace uses a roster management and indexing API to easily manage, view users, and manage Access Control List (ACL) based on a list of users. When there is a change in a group membership of a user, the API will send a notification. The notification is then compiled by the compilation engine and the output of the compilation is sent to a backend setting service which is used by different Workspace apps to determine whether an app is enabled for a given user and to provide users with appropriate access to the app.

A recent rollout to the roster management backend database inadvertently introduced an issue which generated unnecessary notification traffic on user computation. When membership changes are made to a large roster, all of its descendants need to be recomputed, generating a lot of backend notifications that need to be recomputed and updated. The trigger for the issue was a membership change request from a large customer which required compilation of a large number of user changes.

This caused rapid compilation of app settings for the same customer in a very short time interval putting excessive load on database partitions holding the relevant data thus creating a hotspot. The hotspotting caused delays in compilation of user settings across Workspace applications.

Remediation and Prevention:

Google engineers were alerted to the issue by internal monitoring on 07 September 2022 at 09:37 US/Pacific and immediately started an investigation. At 11:57 US/Pacific, engineers identified the hotspotting issue and narrowed down the trigger. At 13:05 US/Pacific, engineers began mitigating the hotspotting by manual intervention. By 13:59, the compilation success rate had improved and started processing the backlog. The backlog gradually progressed and completed for all the affected customers at 19:11 US/Pacific resolving the issue.

Google is committed to quickly and continually improving our technology and operations to prevent a future recurrence by taking the following actions:

• Enhance the compilation scheduling design for better fairness and isolation of customer requests.
• Enhance debugging tools so that on-call engineers can quickly identify the issues from compilation logs.
• Complete rollout to fix the issue with the roster management backend database.

Detailed Description of Impact:

Gmail: Google Workspace users, including newly created users that had Gmail newly enabled for them, between 09:04 US/Pacific and 19:11 US/Pacific on 7 September 2022 experienced a delay before they were able to access Gmail.

Other Workspace apps: Users may have experienced delays in accessing other Workspace apps after those apps were enabled for them. Our analysis did not reveal any noticeable errors for other Workspace apps.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article: https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 07 September 2022 09:04

Incident End: 07 September 2022 23:29

Duration: 14 hours, 25 minutes

Affected Services and Features:

Google Workspace - Gmail

Regions/Zones: Global

Description:

Newly created Google Workspace users experienced a temporary error while accessing their Gmail account for a total duration of 14 hours, 25 minutes. From preliminary analysis, the root cause of the issue is due to a large number of compilation queue messages in one domain which slowed compilation in other domains for other users.

Customer Impact:

New Google Workspace user accounts which were created after 07 September 2022 09:00 US/Pacific or whose domain policies/subscriptions were modified since that time, experienced the following error while trying to access Gmail:

“ We’re sorry, but your account is temporarily unavailable. We apologize for the inconvenience and suggest trying again in a few minutes. You can view the Google Workspace Status Dashboard for the current status of the service. Numeric Code: 7354 “

Affected users may have experienced issues while accessing other Workspace services.

Additional details:

The issue was resolved once the backlog of queued compilation messages was cleared.