Rackspace Incident Highlights How Disruptive Attacks on Cloud Providers Can Be

A Dec. 2 ransomware attack at Rackspace Technology — which the managed cloud hosting company took several days to confirm — is quickly becoming a case study on the havoc that can result from a single well-placed attack on a cloud service provider.

The attack has disrupted email services for thousands of mostly small and midsize organizations. The forced migration to a competitor’s platform left some Rackspace customers frustrated and desperate for support from the company. It has also already prompted at least one class-action lawsuit and pushed the publicly traded Rackspace’s share price down nearly 21% over the past five days.

Delayed Disclosure?

“While it’s possible the root cause was a missed patch or misconfiguration, there’s not enough information publicly available to say what technique the attackers used to breach the Rackspace environment,” says Mike Parkin, senior technical engineer at Vulcan Cyber. “The larger issue is that the breach affected multiple Rackspace customers here, which points out one of the potential challenges with relying on cloud infrastructure.” The attack shows how if threat actors can compromise or cripple large service providers, they can affect multiple tenants at once.

Rackspace first disclosed something was amiss at 2:20 a.m. EST on Dec. 2 with an announcement it was looking into “an issue” affecting the company’s Hosted Exchange environment. Over the next several hours, the company kept providing updates about customers reporting email connectivity and login issues, but it wasn’t until nearly a full day later that Rackspace even identified the issue as a “security incident.”

By that time, Rackspace had already shut down its Hosted Exchange environment citing “significant failure” and said it did not have an estimate for when the company would be able to restore the service. Rackspace warned customers that restoration efforts could take several days and advised those looking for immediate access to email services to use Microsoft 365 instead. “At no cost to you, we will be providing access to Microsoft Exchange Plan 1 licenses on Microsoft 365 until further notice,” Rackspace said in a Dec. 3 update.

The company noted that Rackspace’s support team would be available to assist administrators configure and set up accounts for their organizations in Microsoft 365. In subsequent updates, Rackspace said it had helped — and was helping — thousands of its customers move to Microsoft 365.

A Big Challenge

On Dec. 6, more than four days after its first alert, Rackspace identified the issue that had knocked its Hosted Exchange environment offline as a ransomware attack. The company described the incident as isolated to its Exchange service and said it was still trying to determine what data the attack might have affected. “At this time, we are unable to provide a timeline for restoration of the Hosted Exchange environment,” Rackspace said. “We are working to provide customers with archives of inboxes where available, to eventually import over to Microsoft 365.”

The company acknowledged that moving to Microsoft 365 is not going to be particularly easy for some of its customers and said it has mustered all the support it can get to help organizations. “We recognize that setting up and configuring Microsoft 365 can be challenging and we have added all available resources to help support customers,” it said. Rackspace suggested that as a temporary solution, customers could enable a forwarding option, so mail destined to their Hosted Exchange account goes to an external email address instead.

Rackspace has not disclosed how many organizations the attack has affected, whether it received any ransom demand or paid a ransom, or whether it has been able to identify the attacker. The company did not respond immediately to a Dark Reading request seeking information on these issues. In a Dec. 6. SEC filing, Rackspace warned the incident could cause a loss in revenue for the company’s nearly $30 million Hosted Exchange business. “In addition, the Company may have incremental costs associated with its response to the incident.”

Customers Are Furious and Frustrated

Messages on Twitter suggest that many customers are furious at Rackspace over the incident and the company’s handling of it so far. Many appear frustrated at what they perceive as Rackspace’s lack of transparency and the challenges they are encountering in trying to get their email back online.

One Twitter user and apparent Rackspace customer wanted to know about their organization’s data. “Guys, when are you going to give us access to our data,” the user posted. “Telling us to go to M365 with a new blank slate is not acceptable. Help your partners. Give us our data back.”

Another Twitter user suggested that the Rackspace attackers had also compromised customer data in the incident based on the number of Rackspace-specific phishing emails they had been receiving the last few days. “I assume all of your customer data has also been breached and is now for sale on the dark web. Your customers aren’t stupid,” the user said.

Several others expressed frustration over their inability to get support from Rackspace, and others claimed to have terminated their relationship with the company. “You are holding us hostages. The lawsuit is going to take you to bankruptcy,” another apparent Rackspace customer noted.

Davis McCarthy, principal security researcher at Valtix, says the breach is a reminder why organizations should pay attention to the fact that security in the cloud is a shared responsibility. “If a service provider fails to deliver that security, an organization is unknowingly exposed to threats they cannot mitigate themselves,” he says. “Having a risk management plan that determines the impact of those known unknowns will help organizations recover during that worst case scenario.”

Meanwhile, the lawsuit, filed by California law firm Cole & Van Note on behalf of Rackspace customers, accused the company of “negligence and related violations” around the breach. “That Rackspace offered opaque updates for days, then admitted to a ransomware event without further customer assistance is outrageous,” a statement announcing the lawsuit noted.

Did the Attackers Exploit “ProxyNotShell” Exchange Server Flaws?

No details are publicly available on how the attackers might have breached Rackspace’s Hosted Exchange environment. But security researcher Kevin Beaumont has said his analysis showed that just prior to the intrusion, Rackspace’s Exchange cluster had versions of the technology that appeared vulnerable to the “ProxyNotShell” zero-day flaws in Exchange Server earlier this year.

“It is possible the Rackspace breach happened due to other issues,” Beaumont said. But the breach is a general reminder why Exchange Server administrators need to apply Microsoft’s patches for the flaws, he added. “I expect continued attacks on organizations via Microsoft Exchange through 2023.”