Reaction to Microsoft Exchange 0-day attack

On March 6, Microsoft pointed out vul­ner­a­bil­i­ties in the Microsoft Exchange software. IONOS had already learned of the vul­ner­a­bil­i­ty on March 3 and im­me­di­ate­ly applied updates provided by Microsoft to all Exchange systems it operated itself in order to eliminate the vul­ner­a­bil­i­ties. IONOS systems were not affected by the attack wave.

In this interview with Lead Exchange Engineer, John Barnes, we try to clarify the most important questions on the matter.

We found out about the vul­ner­a­bil­i­ty very early in the morning of March 3rd and we im­me­di­ate­ly began assessing which servers needed to be patched and splitting up the work among the team to expedite the patching process.

We knew im­me­di­ate­ly that this update was very important, because it is rare for Microsoft to release patches outside of their standard monthly process. As soon as we got into the details of the vul­ner­a­bil­i­ties and un­der­stood the technical ram­i­fi­ca­tions it became apparent that this patching needed to happen ab­solute­ly as quickly as possible.

We build our platforms with a sig­nif­i­cant amount of re­dun­dan­cy, so in cir­cum­stances such as these we are able to patch our platforms during business hours without causing a loss of service to our customers. Where possible, this also allows us to automate the patching process so that a sig­nif­i­cant pro­por­tion of our servers had already installed the patch prior to the start of the day.

The other parts of our platform we began patching in stages, moving workload between servers to avoid service in­ter­rup­tion which took most of the day on Wednesday, pri­ori­tis­ing those parts of the platform we felt were most vul­ner­a­ble. We also became vigilant in looking for any sus­pi­cious activity on our platforms.

While the patching work was ongoing, we also joined a call from Microsoft regarding the update and co­or­di­nat­ed with the IONOS Security Team to ensure in­for­ma­tion was dis­trib­uted through­out the wider business. The Microsoft call was par­tic­u­lar­ly en­light­en­ing with how much emphasis was being placed on patching right now, and not trying to mitigate the exploit in other ways.

Once the patching process was complete, we turned our primary focus to scanning for In­di­ca­tors of Com­pro­mise. This is a lengthy process due to the size of our platforms, and how seriously we took these vul­ner­a­bil­i­ties, so we devoted several days to this scanning process.

Patching the vul­ner­a­bil­i­ty as quickly as possible should be the number one priority, which can be difficult sometimes with systems that are business critical such as email. In cases such as these, where the vul­ner­a­bil­i­ty can be so damaging, it is worth con­sid­er­ing whether to cause a service outage in order to deploy the patch faster.

As a hosting provider we are geared up to act quickly and de­ci­sive­ly on these 'in­ci­dents'. As this is something we do every day, we have the contacts and the in­for­ma­tion to know about these things as soon as possible - often before public releases. 

Our processes are highly automated and managed, therefore we ensure that we can react quickly and we operate globally scaled platforms, with geo-re­dun­dan­cy, meaning updates can be applied without business dis­rup­tion.  As a Hoster we have been running global Hosted Exchange platforms since 2010 and have un­ri­valled ex­pe­ri­ence outside of Microsoft.  We un­der­stand that these things happen - and it is how you can react that is most important.

I think the companies most at risk are the smaller companies running Exchange Servers on premise, that don’t nec­es­sar­i­ly have the resources to keep on top of the quarterly Cu­mu­la­tive Update schedule from Microsoft.

When Microsoft initially released the updates they were only available for the latest two Cu­mu­la­tive Updates on each version of Exchange, which meant that companies that were not up to date would need to install the latest Cu­mu­la­tive Updates before they could patch the vul­ner­a­bil­i­ty. This can be a much more involved process than just in­stalling a Security Update for Exchange and will take sig­nif­i­cant­ly longer, in­creas­ing the company’s exposure to this vul­ner­a­bil­i­ty.

For this specific vul­ner­a­bil­i­ty: Microsoft rec­om­mends this HealthCheck­er to find the patch level of your servers and determine if you need to update your servers. In par­tic­u­lar it may not be apparent through Microsoft Update that you have an out­stand­ing update if your Exchange Servers are not patched to the latest Cu­mu­la­tive Update. The Test-Prox­y­L­o­gon script is a Microsoft script which will help you identify any In­di­ca­tors of Com­pro­mise.

In general usage I have found the Nessus Vul­ner­a­bil­i­ty Scanner to be par­tic­u­lar­ly effective at iden­ti­fy­ing vul­ner­a­bil­i­ties and unpatched servers.

This exploit was par­tic­u­lar­ly nasty, an ‘Unau­then­ti­cat­ed Remote Code Execution as System’ exploit. It is par­tic­u­lar­ly dangerous in this case, because Microsoft Exchange Servers typically have high levels of Privilege within Active Directory which is the primary Au­then­ti­ca­tion and Au­tho­ri­sa­tion system for Windows systems within a business.

This means that the risks to the business are extremely high, with a sig­nif­i­cant pos­si­bil­i­ty of stolen/destroyed business data, loss of ability to function etc. It is difficult to un­der­state the possible impact to the business if an attacker was able to suc­cess­ful­ly exploit this vul­ner­a­bil­i­ty.

I think it's always difficult to give a de­fin­i­tive 'all clear' with these sorts of vul­ner­a­bil­i­ties. We've done all the scans, based on the in­for­ma­tion currently provided by Microsoft and we've not seen anything of note, but a really good malicious actor can make it very difficult to spot a com­pro­mise.

In par­tic­u­lar it has now come to light in various news sources that these par­tic­u­lar vul­ner­a­bil­i­ties were first iden­ti­fied in early January and reported re­spon­si­bly to Microsoft. In late February, more wide­spread attacks were iden­ti­fied just prior to Microsoft releasing the patches. This means that there is a sig­nif­i­cant time frame for the initial exploit to occur.

I believe this may make companies reassess whether they wish to continue hosting their email systems on premise and the real cost of main­tain­ing those systems versus hosting these business critical systems with a trusted cloud provider.

Further In­for­ma­tion:

https://www.microsoft.com/security/blog/2021/03/02/hafnium-targeting-exchange-servers/ 

https://msrc-blog.microsoft.com/2021/03/02/multiple-security-updates-released-for-exchange-server/ 

https://msrc-blog.microsoft.com/2021/03/05/microsoft-exchange-server-vul­ner­a­bil­i­ties-mit­i­ga­tions-march-2021/