Google Voice Disruption Caused by Expired TLS Certificates

Google Voice is a Google voicemail service that allows users to send free texts, personalize the voicemail, read text transcripts for voicemail, and much more. The voicemail service of Google, which previously required a Google Voice invitation code for installation, is now free of charge available for all Gmail users. 

The incident report states that, “Google Voice users experienced an issue in which some new inbound or outbound Voice over Internet Protocol (VoIP) calls failed to connect, for a total duration of 4hours 22 minutes.” 

In order to manage phone calls over the Internet protocol, Google Voice uses the Initiation Session Protocol (SIP). Google Voice consumer devices aim at ensuring a continuous SIP link with Google Voice services during routine operation. The customer tries to regain contact automatically after a link fails. Transport Layer Security (TLS) certificates are also rotated periodically to ensure that all Google Voice traffic is protected and linked. 

“Due to an issue with updating certificate configurations, the active certificate in Google Voice frontend systems inadvertently expired at 2021-02-15 23:51:00, triggering the issue,” Google explained. “During the impact period, any clients attempting to establish or re-establish a SIP connection were unable to do so.” 

Users could not access the Google Voice platform to make or accept VoIP calls following the breakdown of expired certificates. However, consumer systems with an active SIP connection were not impacted during the outage before this incident (as long as the connection was not interrupted). The technical team concluded after the analysis that the root cause was certificate configuration. The team has developed and initiated an emergency roll-out of modified credentials and configuration information to interfaces. After mitigation was enforced, the functionality of Google Voice SIP customers restored retrieval of their connections.

Publishing the incident report, the Google Workspace Team stated the steps taken by the engineers. They insisted on, setting additional constructive warnings for credential expiry incidents to come, and set up additional reactive warnings in Google Voice frontend applications for TLS errors. Alongside, enhance automatic credential rotation tooling and changes to set up and to allow the quick rollout of configuration improvements, utilizing more portable facilities. Developing emergency roll-out testing and practice examples with Google Voice interface applications and settings.

Google is committed to improving our technology and operations efficiently and consistently to avoid service disruptions. They said that “We thank your patience and excuse your company for any effects. For your company, we appreciate you.”