3 security lessons from an MSP that survived the Kaseya VSA attack

Jay Tipton, chief executive for the Managed Service Provider (MSP) Technology Specialists, remembers his Fourth of July weekend this year like many MSP employees likely remember theirs: As a bit of a nightmare.

“That’s like the worst feeling you’ll ever have,” Tipton said about his initial impressions about a fast-moving ransomware attack that he originally thought hit just his company. His Microsoft Outlook instance closed down unexpectedly, his phone rang and he learned about a customer having trouble connecting to some software tools, and then, just minutes later, his phone rang again. The number of customer problems had already multiplied.

As Tipton and the world would soon learn, his Fort Wayne, Indiana-based MSP was just one of up to 1,500 companies ensnared in what was is probably the largest ransomware attack ever, when threat actors poisoned the remote monitoring and management software tool Kaseya VSA—a favorite for many MSPs—with ransomware.

The attack, which actually led to grocery stores shuttering their doors in Sweden, proved so detrimental because of its cascading nature. By attacking Kaseya VSA, threat actors not only managed to compromise the software, but also the MSPs that used the software, and the small- to medium-sized businesses that were supported by those same MSPs.

Recovery for Tipton’s company has been slow but hopeful. Technology Specialists retrieved data for its customers, maintained strong customer relationships, and even received an outpouring of support from ex-employees and clients themselves.

But in speaking with MJ Shoer, executive director for the nonprofit CompTIA’s Information Sharing and Analysis Organization, Tipton revealed that even the best recovery plans will hit unforeseen obstacles.

Take, for instance, Technical Specialists’ efforts in recovering their clients’ data. Their backups worked, Tipton said, but the process itself happened slower than expected.

“We’ve had some restoring issues, and part of it had to do with download speeds, because everyone was trying to hit the same data centers at the same time,” Tipton told Shoer. “That’s part of the problem. You can’t plan for that.”

Through this process, Tipton compiled a long list of things he’d like to change moving forward, most of it on a large Post-It note covering much of one of his walls. Here’s what Tipton is focusing on moving forward. His lessons are relevant to all organizations, not just MSPs.

Ransomware recovery lessons

1. Put passwords and disaster recovery plans on paper

If the worst happens, you’ll wish you had made a recovery plan. Recovery plans typically identify the key systems and data inside your organization, and the shortest path to restoring critical business functions.

Following the Kaseya VSA ransomware attack, Tipton said that he is focusing on a way to provide “paper printouts” for his company and his clients’ disaster recovery plans. He also added that he wants to find a way to “securely print out passwords” because the attack also seemingly affected Technical Specialists’ password vault.

“We had to wait almost 36 hours to get our password vault restored so we could get passwords out of it,” Tipton said.

Both ideas have immediate value for any business, big or small. A disaster recovery plan is only as useful as it is accessible, and an inaccessible password vault could slow down literally every single part of a data recovery effort if administrators simply cannot access their accounts.

2. Say goodbye to public whitelists

Allowing MSPs to manage some or all of their IT and security makes sense for lots of small businesses, but it comes with its own risks. MSPs act as administrators, so any tools they use get administrator privileges too. MSPs also need to make their toolchain work across all the various customer environments they work with too.

A common practice for MSP software vendors is to advise users of directories that should be “whitelisted” against antivirus software, so that their software can work without interference from cybersecurity tools. This practice is understandable—attackers try hard to disguise themselves as administrators and security tools have the difficult job of letting legitimate remote administration go ahead while stopping malicious remote administration—but it is ill-advised.

These whitelist guides are available for anyone to view online, but, according to Tipton, Technical Specialists is asking for more control into how to actually treat some directories. Tipton said some of what he’s doing moving forward is “not allowing the software vendors to push us into whitelisting directories. That’s not happening anymore.”

“Give me control of which directory it is and how far down I can bury it—I’ll consider it, because then I can control how it’s working, what’s going on in there, and where it’s at so it’s not public knowledge that directory exists,” Tipton said. “But this open whitelisting of programs and directories isn’t going to happen.”

3. Insist that software is digitally signed

In speaking with Shoer, Tipton mentioned that one of the vendors that Technical Specialists use has the annoying habit of changing its DLLs (the software libraries that their product uses) quite regularly. Tipton said he will not allow that anymore unless the vendor starts digitally signing the DLLs.

Why? Because this is another situation where legitimate behavior and malicious behavior can look very similar. If a DLL changes and it hasn’t been signed by the vendor, Tipton has no way of knowing if the new DLL is legitimate or if it has been tampered with by an attacker.

“I’ve got a vendor that likes to keep changing their DLLs, and I think some of them change on the fly and it causes all kinds of problems,” Tipton said. “You’re going to have to sign your program with a cert because I’m going to block it and it’s not optional.”

Moving on

People are often understandably reluctant to talk about their experiences with ransomware, so we applaud Tipton for being open and transparent, and giving us all the opportunity to benefit from his experience.

All of Tipton’s goals seem to be focused on giving Technical Specialists more visibility and capability into how it supports its clients. And perhaps that’s the right mindset—Tipton shared with Shoer that his business lost very few clients after the attack, and of the clients he did lose, seemingly all of them misplaced blame on the MSP itself.

“There are a few that don’t get it, won’t ever get it, will never understand, and say it’s all our fault,” Tipton said. “I can’t change their minds, so I’ll just shake their hands, part as friends, and go on with life.”

Ransomware podcasts

Ransomware recovery is an important subject that benefits enormously from the real-world perspective and experience of those who have been through it. Several recent episodes of Malwarebytes Labs’ Lock and Code podcast have dealt with different aspects of recovering from ransomware.

Racing against a real-life ransomware attack

At 11:37 pm on the night of September 20, 2019, cybercriminals launched a ransomware attack against Northshore School District in Washington state. Early the next morning, Northshore systems administrator Ski Kacoroski arrived on scene. Kacoroski explains what happened next, and what Northshore did to recover from the attack and prevent it from happening again.

Listen to Racing against a real-life ransomware attack

“Seven or eight” zero-days: The failed race to fix Kaseya VSA

The Dutch Institute for Vulnerability Disclosure (DIVD) discovered “seven or eight” zero-days in Kaseya VSA before the REvil ransomware group did. DIVD chair Victor Gevers explains why that wasn’t enough to stop the biggest ransomware attack in history, and reveals that Kaseya VSA’s vulnerabilities represent just one data point in a far larger and more worrying trend.

Listen to “Seven or eight” zero-days: The failed race to fix Kaseya VSA

Why backups aren’t a “silver bullet” against ransomware

Any cybersecurity expert will tell you that the last line of defense against ransomware is backups. But if they’re so important, why are we still so bad at getting them right? Host David Ruiz speaks with VMware’s Matt Crape about why making good backups is so hard, and what missteps you should watch out for.

The post 3 security lessons from an MSP that survived the Kaseya VSA attack appeared first on Malwarebytes Labs.