The Cloud and High Availability

One of my new favorite podcasts is RunAsRadio with Richard Campbell. The June 14, 2023 edition was "High Availability in 2023 with Allan Hirt".

You can see Allan Hirt's credentials on LinkedIn.

Here's the synopsis of the podcast:

What does high availability look like in 2023? Richard chats with Allan Hirt about his work with high-availability solutions today - not just on-premises but also in the cloud. Allan talks about the frustration folks had with moving workloads in the cloud during the pandemic panic, lift-and-shifting workloads focusing on getting things working quickly rather than cost-effectively. The results can be costly, to the point where some folks considering moving back off the cloud again - but does that make sense? Allan talks about creating high availability efficiently wherever you want to run your workloads!

Richard and Allan covered availability as manifested in the cloud. Allan is a SQL Server guru who works for Pure Storage.

Here are some excerpts.

7:31 But when it comes to high availability, really, it's tough.

And cloud isn't a silver bullet for availability.

7:58 Like everybody assumes that there's this big red button that says I've got high availability in any cloud. We're not just talking Azure, we're talking AWS and GCP.

And the cloud doesn't eliminate traditional availability issues.

8:54 You know, like your platform layer, meaning Azure or GCP, like I can spin up this VM in another region when something happens is great. But the problem then winds up being it doesn't account for things like transactions in a SQL Server installation. So all the traditional things we worry about on premises still exist up in the cloud with IaaS. It's PaaS where things get different.

And the cloud introduces new issues.

13:15 … but now we need to think about placement of our VMs. We need to think about like ensuring that the VMs don't sit on the same host or in the same rack, which means setting certain things differently in your cloud providers.

And old issues don't go away.

15:10 … I appreciate the idea that if you're responsible high availability, … that doesn't go away with the cloud. … You still are responsible for [high availability] and you have to work through the additional complexity.

And monitoring gets even more important.

23:11 And you really don't have a view of is it functioning well. And then you need somehow you need to be able to monitor the stuff remotely and realize we have no insight into what's going on into our systems. I saw an uptick in implementation of like a lot of the third party monitoring tools or SCOM … because people "Like we need this stuff." Whereas they didn't put too much importance on that.

But cloud providers likely provide higher availability than you could.

28:15 I still think the cloud is up more than any data center I ever ran was. What I've actually told customers is Microsoft, Amazon, and Google can outspend you in building a data center. And a 24 hour staffed NOC is normal for them and expensive for anybody else.

But just moving to the cloud doesn't guarantee more benefits.

30:46 So I can take my on premises VMware infrastructure, vMotion it up to Azure, AWS, GCP and vice versa all day long, right. … I think for some folks that's a good solution. … It gets you to the cloud. It works well sort of. … And so it's always interesting to see things like this is where it's more of a on premises play in the cloud.

If you're not good at on premises, you won't be any better on the cloud.

34:56 If you sucked at virtualization on premises, you're generally going to suck at stuff in the cloud. Not always, but it's generally true because if you couldn't size and you couldn't deploy well on premises, how are you going to automate and do all this cool dev ops, dev stack ops stuff. If you had no skills to do it on premises, it's not magically going to happen in the cloud.

AWS - Amazon Web Services

GCP - Google Cloud Platform

IaaS - Infrastructure as a Service

PaaS - Platform as a Service

SCOM - Microsoft System Center Operations Manager

NOC - Network Operations Center

You Need a Side Channel

Here we Microsoft go again! There's a list of Microsoft availability problems here. Don't think I'm all doom and gloom on Microsoft. It's just that even HUGE organizations struggle with subtleties.

Microsoft's latest incident affected Azure, Teams, and Outlook for hours.

Microsoft recently released their postmortem. I applaud Microsoft for publishing this. Could your company have done such a thorough job so quickly?

As part of a planned change to update the IP address on a WAN router, a command given to the router caused it to send messages to all other routers in the WAN, which resulted in all of them recomputing their adjacency and forwarding tables. During this re-computation process, the routers were unable to correctly forward packets traversing them.

Maybe your network isn't so large that this "re-computation process" wouldn't saturate your network equipment.

Regardless there is a learning here.

Due to the WAN impact, our automated systems for maintaining the health of the WAN were paused, including the systems for identifying and removing unhealthy devices, and the traffic engineering system for optimizing the flow of data across the network.

Their network management system, including device security,  ran ACROSS their network. So when the network was impacted their network management system was ineffective. Basically, Microsoft had to watch and wait for the network to settle down.

A side channel network management solution would have mitigated that. And introduced a myriad of other problems, principally security.

Tough choices.

Windows 10 21H2 Enablement Package

Windows 10 November 2021 Update, a.k.a. 21H2, doesn't have many new features, especially for the home user.

• Adding WPA3 H2E standards support for enhanced Wi-Fi security
• Windows Hello for Business introduces a new deployment method called cloud trust to support simplified passwordless deployments and achieve a deploy-to-run state within a few minutes
• GPU compute support in the Windows Subsystem for Linux (WSL) and Azure IoT Edge for Linux on Windows (EFLOW) deployments for machine learning and other compute intensive workflows

It hasn't shown up on ANY of my PCs so far. I always try to test new things so you don't have to. 21H2 is no exception.

Like 21H1, I found a link to download the enablement package. And like 21H1, when I clicked on that link, nothing happened. I had to right click and choose "Copy link address" and then paste that link into another browser tab.

Worked fine.

A Fool and His Data Are Soon Parted

Azure just keeps having problems. And a recent incident happened while their CEO was speaking at a conference in Sydney "focused on showing off how Microsoft's cloud technology can have a transformative impact on businesses."

While it's easy to pile on with Azure, the situation is endemic to the cloud environment. Not withstanding my recent post, cloud implementations still carry significant risks. The news.com.au article states:

While cloud technology can have a massive impact on the way businesses run, as we are currently witnessing, it can also create new headaches when things don’t work how they’re supposed to.

And this was the second Microsoft cloud outage in 24 hours.

Really worrisome to me is that the Microsoft Office portal wasn't responsive.

As usual the comments are the most interesting. Clearly some are trolls but some are really to the point.

captain goodvibes
Anyone who trusts cloud technology to store data risks it disappearing into thin air. The only way to safely store data is on a hard drive. To paraphrase an old adage, a fool and his data are soon parted.

Anthony
The problem is for a lot of organisations is that they are too small to safely and economically run their ICT. Large vendors can provide resilience, security, and responsive services easier. And no, I don’t work for Microsoft, but I have worked in outsourcing :-) so I know the numbers

HG
If your business model is a monopoly, make sure your cr*p works.

Gordon
I spent most of my career in IT, I just do not understand why anyone would trust another company to manage all of their business critical data off site....

Yeah, worry about your business critical data.

But Can You Do Better Than This

If you've been reading this blog, you know that I worry about cloud availability.

However, most of the time the big cloud providers have more availability and redundancy capability than almost any enterprise can provide.

For an example, Microsoft recently had an outage of its Multifactor Authentication (MFA) for Azure and Office 365 users in North America,

There's a report on it here. And here's Microsoft's Root Cause Analysis.

Go read the Next Steps. Here's #1:

1. Fine-grained fault domain isolation work has been accelerated. This work builds on the previous fault domain isolation work which limited this incident to North American tenants. This includes:

• Additional physical partitioning within each Azure region.
• Logical partitioning between authentication types.
• Improved partitioning between service tiers.

Do you think your enterprise could add "Additional physical partitioning," "Logical partitioning between authentication types," and "Improved partitioning between service tiers?"

If you could, you must be in the Fortune 100. And that's just the first bullet of actions.

These outages from the cloud providers get a lot of publicity but unless you're a mega-scale enterprise any of them are way more capable than you'll ever hope to be.

The Cloud is Full

Recently ZDNet's  All About Microsoft reported that Microsoft Azure customers had reported hitting virtual machine limits in U.S. East regions.

This is scary.

There were a small number of comments (7) that related that they had had the same experience. Here is a typical comment.

GetNrDone
Happened to me. I tried to deploy a new SQL database in eastus2 2 weeks ago only to be greeted by an error. Opened a tickets and was basically told there was nothing they could do. Escalated the issue with our TAM which also could not get approval for 1 database to deploy. I was asking for the smallest database they offer (s0) and was told we couldn't have it. No communication before, no warning emails, no blog posts, nothing in the service dashboards, even our account team didn't know anything about it. Completely blindsided me and delayed development on an app for a week while i could move resources to another region. Unacceptable and definitely does not live up to promises made!

There's a reddit thread on this here.

dops0
We've faced this issue in North Europe, East US and West US 2. This has just started happening over the last couple of weeks and what's even more frustrating is, we already have sufficient quota allocated to us, but, our users haven't been able to deploy their machines even when within this quota.

This issue hasn't blown up so either it has been resolved or customers were able to work around it as GetNrDone did.

Regardless this is another consideration for using anyone's cloud services. I'd even suggest trying to putting a clause in your contract that guaranteed x% of available capacity on the vendor's part. I'd bet that none of the vendors would accept that but that would at least make the vendor play their cards.

2019-07 Cumulative Update for Windows 10 Version 1903

In late July Microsoft released a new cumulative update (the second one in July) for the Windows 10 version 1903 that includes a series of quality improvements.

Bleeping Computer has a thorough article (archive.is) on it.

It is an optional update and you have to click on a new "Download and install" option on Windows Update settings page to apply the patch.

It all worked fine for me but the list of "quality improvements" overwhelmed me.

• Addresses an issue that fails to record a local user’s last sign in time even when the user has accessed the server’s network share. 
• Addresses an issue that may sever the domain trust relationship when you enable Recycle Bin in the domain that established the trust relationship. 
• Addresses an issue that prevents Windows Hello face authentication from working after a restart. 
• Updates time zone information for Brazil. 
• Addresses an issue to enable Microsoft Edge to print PDF documents that contain landscape and portrait-oriented pages correctly. 
• Addresses an issue with PDFs that are configured to be opened only once in Microsoft Edge. 
• Addresses an issue that may display colors incorrectly when viewing an image on 10-bit display panels. 
• Addresses an issue that may prevent you from changing the display brightness when using certain graphics drivers after resuming from Sleep or Hibernation. 
• Addresses an issue in which Windows Graphics Device Interface (GDI+) returns an empty font family name for Bahnschrift.ttf. 
• Addresses an issue that may cause a mouse press and release event to sometimes produce an extra mouse move event. 
• Addresses an issue that may cause the UI to stop responding for several seconds when scrolling in windows that have many child windows. 
• Addresses an issue that fails to bypass automatic sign in (Autologon) when you press and hold the Shift key during startup. 
• Addresses an issue that may prevent a device from going to Sleep mode when certain applications that rely on Bluetooth are open. 
• Addresses an issue that may reduce Bluetooth audio quality when using certain audio profiles for extended periods. 
• Addresses an issue that prevents Microsoft Application Virtualization (App-V) scripting from working if you run it when you’re not connected to a domain controller (DC). App-V scripting also fails when you run it in an environment that only contains Microsoft Azure Active Directory. 
• Addresses an issue that causes an error if you open Microsoft OneDrive files on demand when User Experience Virtualization (UE-V) is enabled. To apply this solution, set the following DWORD to 1: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\UEV\Agent\Configuration\ApplyExplorerCompatFix 
• Addresses an issue with UE-V that may sometimes prevent exclusion paths from working. 
• Addresses an issue that may cause a system that has a Trusted Platform Module (TPM) to stop working. 
• Addresses an issue that prevents a system from recognizing a Microsoft account or Azure Active Directory account until the user signs out and signs in again. 
• Addresses an issue that may prevent the Netlogon service from establishing a secure channel and reports the error, “0xC000007A – ERROR_PROC_NOT_FOUND.” 
• Addresses an issue that fails to update the personal identification number (PIN) policy (minimum length, required digits and special characters, etc.) for Windows Hello for Business when a PIN already exists on the machine. 
• Addresses an issue that causes failures to create a recovery drive (USB key) with the error, ”0x80042405[gle=0x00000715].” 
• Addresses an issue that prevents an Android emulator based on the virtual machine platform from starting on some systems. 
• Addresses an issue that uses a temporary profile to sign in a local user account when the account is configured with a mandatory roaming user profile. The error, “We can’t sign in to your account” appears. The Application event log contains Event 1521, and the event’s source is listed as the Microsoft-Windows-User Profiles Service. 
• Addresses an issue that changes the status for Work Folders in File Explorer to 0x80C802A0 (ECS_E_SYNC_UPLOAD_PLACEHOLDER_FAILURE) after selecting Free up space. 
• Addresses an issue that may cause a Remote Desktop Server to stop responding when someone who is using drive redirection disconnects. 
• Addresses an issue that may cause the Remote Access Connection Manager (RASMAN) service to stop working. You may receive the error “0xc0000005” on devices that have the diagnostic data level manually configured to the non-default setting of 0. You may also receive an error in the Application section of Windows Logs in Event Viewer with Event ID 1000 referencing “svchost.exe_RasMan” and “rasman.dll”. This issue only occurs when a virtual private network (VPN) profile is configured as an Always On VPN (AOVPN) connection with or without a device tunnel.
• Addresses an issue that causes applications on a container host to intermittently lose connectivity because of a port conflict with applications running on a container.
• Addresses an issue that intermittently prevents connections to a corporate network when using Always On VPN with the IKEv2 protocol. Connections are not always automatically established, and manual connections sometimes fail. In this scenario, when you call the RasDial function from the command line for the target VPN connection, you receive the error, “ERROR_PORT_NOT_AVAILABLE(633)”.
• Adds limited support for Windows voice dictation for Chinese Simplified, English (Australia, Canada, India, United Kingdom), French (France), German (Germany), Italian (Italy), Portuguese (Brazil), and Spanish (Mexico, Spain).
• Addresses an issue with opening or using the Window-Eyes screen reader application that may result in an error and prevent some features from functioning as expected.
• Addresses an issue that prevents an App-V application from opening and displays a network failure error. This issue occurs under certain circumstances, such as when a system's battery is low or there is an unexpected power failure.
• Addresses an issue that prevents an App-V application from opening if the client is offline and a startup script is defined for the App-V application.
• Addresses a rare issue that causes Windows Defender Advanced Threat Protection (ATP) to temporarily prevent other processes from accessing files.
• Addresses an issue that causes the Start menu to stop responding when new users sign in to Windows 10, version 1903.
• Updates the Windows Ink Workspace by simplifying the menu and adding direct integration with the Microsoft Whiteboard app for a richer collaboration experience.

Then there's a list of known issues in this update.

• Windows Sandbox may fail to start with "ERROR_FILE_NOT_FOUND (0x80070002)" on devices in which the operating system language is changed during the update process when installing Windows 10, version 1903.
  
  Microsoft is working on a resolution and will provide an update in an upcoming release.

• Devices that start up using Preboot Execution Environment (PXE) images from Windows Deployment Services (WDS) or System Center Configuration Manager (SCCM) may fail to start with the error "Status: 0xc0000001, Info: A required device isn't connected or can't be accessed" after installing this update on a WDS server.
  
  For mitigation instructions, see KB4512816. We are working on a resolution and will provide an update in an upcoming release.

• Devices connected to a domain that is configured to use MIT Kerberos realms may not start up or may continue to restart after installation of this update. Devices that are domain controllers or domain members are both affected.
  If you are not sure if your device is affected, contact your administrator. Advanced users can check if this registry key exists HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System\Kerberos\MitRealms or for “Define interoperable Kerberos v5 realm settings” policy under Computer Configuration -> Policies -> Administrative Templates > System -> Kerberos.
  
  At this time, we suggest that devices in an affected environment do not install this update. We are working on a resolution and estimate a solution will be available in mid-August.

Scary.

Friends Don't Let Friends

Microsoft Azure has had another round of problems.

Microsoft Office 365 users in Europe unable to access mailboxes for a full day

Microsoft cloud services see global authentication outage

Microsoft threw Level 3 under the bus on the second outage.

On the Azure status page, Microsoft indicated that the source of the problem is with Level 3, an US-based ISP that provides connectivity and various other services to Microsoft data centers.

At least customers could get to the Azure status page this time.

So I went of to downdetector.com. Here's what they showed several hours after the problem seemed to be resolved.

1600 incidents reported to downdetector.com at the peak.

And impacting several continents.

But Level 3 (CenturyLink) is a backbone provider. Shouldn't a Level 3 problem have affected many service providers?

So back to downdetector.com. Here's what they reported for Amazon Web Services (AWS).

No apparent correlation to the time of the Azure outage and less than 10 incidents reported in the worst hour.

So it must not be a backbone issue impacting multiple service providers?

Not so quick.

AWS' incidents were from the same geography as Azure's.

My assessment:

• Level 3 (or some backbone provider) probably had an incident that affected all its customers.
• AWS was able to tolerate the backbone outage better than Azure.

YOU get to decide where you want to put YOUR data.

Cloud - Office 365

Sometimes it's just too easy to make fun of a company.

In May 2017 Microsoft had gathered their favorite developers in Redmond for the annual Build conference. The focus of that conference was Microsoft's offerings of Office and Azure.

Yeah, you guessed it. Office 365 went down in the middle of the conference as documented by the Register.

Outages happen to everybody.

What I thought interesting were the comments on the Register post:

Cloud just means somebody else's computer that you have NO control over, and can go down at any time leaving you helpless.
Herby

Cloud: A computing service that signifies you do not know where any outage is, who is responsible for it, who else but you or your organization accesses the data on it, but nobody wants to do without.
Jeroen Braamhaar

Having worked with Cloud products for near 10 years now, I can say that in general they are much more reliable and lower cost than an on prem equivalent. ... [Y]ou do get a much better infrastructure from a company that invests millions/billions and has SLA's around it all than the bit of tin you got from the reseller round the corner.
Babbit55

This situation is kinda like when I was on a conference call with my CIO and his boss the CFO. It wasn't a pleasant discussion. When the call was over my CIO turned to me and said "I don't ever want to have that discussion again."

Don't find yourself or your company in the situation that Microsoft was in. Consider and act on the points in those comments.

Cloudy Day

To some I'm sure that March 1st felt like April 1st. Really, that couldn't be happening? Amazon's S3 (Simple Storage Service) went down in their Eastern Region (Ok, it just had "high error rates").

But there are a couple of lessons to be learned from this.

First, it seems nobody is listening to me.

Cloud services aren't magical (even Apple's). They rumble. They go bump.

Don't abdicate your responsibilities to the cloud provider. If you need high availability make sure that that is what your contract guarantees.

In the March 1st S3 outage either lots of customers didn't feel they needed high availability or they misunderstood what they contracted for.

Make sure you are not surprised like "Docker's Registry Hub, Trello, Travis CI, GitHub and GitLab, Quora, Medium, Signal, Slack, Imgur, Twitch.tv, Razer, heaps of publications that stored images and other media in S3, Adobe's cloud, Zendesk, Heroku, Coursera, Bitbucket, Autodesk's cloud, Twilio, Mailchimp, Citrix, Expedia, Flipboard, and Yahoo! Mail (which you probably shouldn't be using anyway)." (source)

At the same time don't over buy. One of my customers was considering migrating their on-premise servers to Azure. As part of their on-premise setup they had a specific backup system and service. When I investigated Azure's service commitments I found that Azure's committed backup and availability met my customers needs and the customer could discontinue their backup system and service.

By the way Amazon did a thorough post mortem on the outage.

Second (and more concerning since they should know better), even Amazon had highly visible services down.

Reminiscent of one of Microsoft's outages Amazon's own online public dashboard was down along with many of Amazon's customer facing services, e.g. Amazon Fire tablets.

From Amazon's post mortem "we have changed the SHD administration console to run across multiple AWS regions."

Amazon, hadn't you thought of this before? What else have you overlooked?

Google Cloud Outages

I've written numerous times about cloud availability. Last year I posted about an outage in Microsoft's Azure service.

Some of my comments were:

Recently there was an 11 hour outage of Microsoft's Azure storage services.

Again users were hard pressed to get details on the outage as "the Service Health Dashboard and Azure Management Portal both rely on Azure."

Cloud outages don't have to be like that.

Here're the communications from Google on a pair of recent outages in their cloud services.

Gmail Outage

November 3, 2015 1:21:00 AM PST

We're investigating reports of an issue with Gmail. We will provide more information shortly.

November 3, 2015 1:38:00 AM PST

Our team is continuing to investigate this issue. We will provide an update by November 3, 2015 2:38:00 AM PST with more information about this problem. Thank you for your patience.

This issue is affecting IMAP and SMTP delivery

November 3, 2015 1:48:00 AM PST

Our team is continuing to investigate this issue. We will provide an update by November 3, 2015 2:48:00 AM PST with more information about this problem. Thank you for your patience.

This issue is affecting incoming POP, SMTP and IMAP connections.

November 3, 2015 2:25:00 AM PST

The problem with Gmail should be resolved. We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better.

Four posts in just over one hour.

Google Calendar Outage

November 4, 2015 8:20:00 AM PST

We're investigating reports of an issue with Google Calendar. We will provide more information shortly.

November 4, 2015 9:25:00 AM PST

Google Calendar service has already been restored for some users, and we expect a resolution for all users within the next 1 hours. Please note this time frame is an estimate and may change.

November 4, 2015 10:25:00 AM PST

The problem with Google Calendar should be resolved. We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better.

Three posts in just over two hours.

Everybody is going to have outages. But this is the way to handle them.

Cloud Reliability Showdown

I keep following enterprise cloud technology and services. Ben Kepes is one of the bloggers I follow. He had a recent article analyzing a report from CloudEndure that compares the reliability of Amazon's AWS.to Microsoft's Azure in 2014.

The article is full of graphs and good analysis but his summary hits the nail on the head:

Outages happen - there is no way of avoiding mentioning that fact. But as Netflix has so successfully demonstrated, it is through planning for failure that organizations can achieve the very highest levels of reliability for their applications. Thinking about multiple points of redundancy for every service, every connection and every vendor an application uses is a good start.

See my article on Netflix.

And the he puts a point on it:

Both services are great and both show amazing reliability (as an aside, that reliability is far higher than most on-premises infrastructure) - the key takeaway from this report is that not planning for failure is a huge risk for organizations.

Put that in your pipe and smoke it.

Enterprise Cloud Providers

As you know, I'm a big follower of cloud technology. Early this year I came across an article in the Wall Street Journal.

In this article the WSJ reported on a recent survey by Piper Jaffray of 112 CIOs. Piper Jaffray asked the CIOs to name their preferred public cloud provider.

Amazon Web Services led the responses with a 2 percentage point increase from 33% to 35%. Microsoft's Azure increased from 20% to 21%. IBM Softlayer had the biggest gain increasing from 4% to 6%.

The big loser was Google falling from 12% to 7%.

Reminds me of the old saying that "Nobody got fired for choosing IBM." Or Microsoft.

There's an interesting comparison of Amazon and Microsoft's cloud offerings here.

More Lessons from the Cloud

There seems to be a recurring theme in my posts about outages in cloud services. While not trying to beat that dead horse there are certainly some lessons to be learned here.

Recently there was an 11 hour outage of Microsoft's Azure storage services.

Again users were hard pressed to get details on the outage as "the Service Health Dashboard and Azure Management Portal both rely on Azure."

I commend Microsoft for owning up to the root problem quickly and succinctly.

"Unfortunately the issue was widespread, since the update was made across most regions in a short period of time due to operational error, instead of following the standard protocol of applying production changes in incremental batches."
     http://azure.microsoft.com/blog/2014/11/19/update-on-azure-storage-service-interruption/

One of the comments summed it up best:

Overdependence.

So much tied into itself that there is no dependency tree - it is a pure network - thus issuing bad changes take down the entire net.

It can be a spectacular update process - minimum to no outage... but only if the updates work.

It also shows a major vulnerability. That central update can take down the entire company if it gets penetrated.

jessepollard
20 November, 2014 12:46

So, lessons...

1. Diversify - Don't build your notification tool on top of what you're monitoring.
2. Manage change - Don't let operational error bite you in the a**. Your execution has to be perfect. Users are unforgiving.

These don't apply just to cloud solutions. They apply just as much to your internal solutions.

My previous posts on this topic:

Storm Clouds
When Clouds Go Bump
When Clouds Go Thump
Lessons from the Cloud
When Clouds Go Bump Revisited
To Be Fair
To Be Fair, Again
To Be Fair, Again and Again

Update: Microsoft has published a thorough analysis of the problem with the corrective action. Good job Microsoft!

To Be Fair

To be fair, Google's Drive service went down recently as did Microsoft's Azure storage service.

To be fair, this is what I got when I went to Microsoft's Dashboard:

To be fair, this is what I got when I went to Google's Dashboard:

To be fair, Microsoft's lasted 12 hours compared to Google's 2+ hours.

To be fair, both probably had lingering effects.

To be fair, this is what life in the cloud is going to be like for the foreseeable future.

Lessons from the Cloud

While researching the recent Microsoft Azure Storage outage I came across a presentation by Jason Chan of Netflix. While the topic was "Practical Cloud Security" there were a couple of slides that I thought applied to many organizations' situation.

We all remember how Netflix changed their business model several years ago going from mailing DVDs to streaming video online. This resulted in a 37 fold increase in API requests in one year.

Realistically most organizations don't experience that kind of growth but any organization that plans to grow should look for lessons learned.

Prior to the cloud migration Netflix was running their own data centers. Here's what they were doing in their data centers.

Sound familiar?

I especially like the "snowflake phenomenon" metaphor. That means that everything is a one-off. Nothing is reused.

Netflix's vision was to get out of the data center business and into the cloud.

We want to use clouds, not build them.

Their target patterns were much different.

Think about how this transition will reduce your costs and improve your delivery speed.

Maybe you can stream a movie in your spare time.

When Clouds Go Thump

Almost a year ago Microsoft's Azure product had a global service failure.

It's happened again.

On the evening of February 22, 2013 Microsoft's Azure Service Dashboard had the following notice:

Storage is currently experiencing a worldwide outage impacting HTTPS operations (SSL traffic) due to an expired certificate. HTTP traffic is not impacted. We are validating the recovery options before implementing them. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

The color is Microsoft's.

Most of the time when I went to the Azure Service Dashboard this is what I got.


Here's what triggered it.


Source: Windows Azure Forum on MSDN
MJFara, one of the posters (identified as a "Partner"), hit the nail on the head.

This is unacceptable, I'm supposed to release an enterprise app on this platform?
Imagine how many phone calls I would have gotten by now from very angry customers.
Sad...

The Register's summary:

It is the opinion of The Register that to have a core service fail in every data center across the world simultaneously is an extremely bad thing to happen to a cloud provider.

From my posting last year:

What I want to highlight is that cloud providers are not immune from service failures. They are likely capable of providing more redundant and resilient services than many organizations can provide.

Unfortunately this failure demonstrates that Microsoft doesn't seem to be "capable of providing more redundant and resilient services."

Go back and reread MJFara's comments. Think about it.

When Clouds Go Bump

There certainly has been a lot of discussion at the CIO level about Cloud Computing. A recent CIO I spoke with was looking at the cloud to help him address his challenges in staffing and running his data center. He and I had deep discussions about his in house alternatives to the cloud. Interestingly while we were having these discussions one of the biggest clouds went bump.

On February 29, 2012 Microsoft's Azure product had a global service failure. Microsoft's Azure Service Dashboard was overwhelmed and pretty much unavailable for hours on end. When it was accessible here is one of the updates approximately 8 hours into the outage:

The restoration steps to mitigate the issue are still underway. This incident impacts Access Control 2.0, Marketplace, Service Bus and the Access Control & Caching Portal in the same regions where Windows Azure Compute is impacted. As a result affected customers may experience a loss of application functionality. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

This is not a Microsoft bashing. In April, 2011 Amazon also suffered a broad failure.

What I want to highlight is that cloud providers are not immune from service failures. They are likely capable of providing more redundant and resilient services than many organizations can provide.

My question is as a CIO are you comfortable with this level of opacity to service failures? Are you willing to answer your users and executives with one-way information flow from a (sometimes available) web page? Would a 10 day service credit make your CEO happy?