The Cloud and High Availability

One of my new favorite podcasts is RunAsRadio with Richard Campbell. The June 14, 2023 edition was "High Availability in 2023 with Allan Hirt".

You can see Allan Hirt's credentials on LinkedIn.

Here's the synopsis of the podcast:

What does high availability look like in 2023? Richard chats with Allan Hirt about his work with high-availability solutions today - not just on-premises but also in the cloud. Allan talks about the frustration folks had with moving workloads in the cloud during the pandemic panic, lift-and-shifting workloads focusing on getting things working quickly rather than cost-effectively. The results can be costly, to the point where some folks considering moving back off the cloud again - but does that make sense? Allan talks about creating high availability efficiently wherever you want to run your workloads!

Richard and Allan covered availability as manifested in the cloud. Allan is a SQL Server guru who works for Pure Storage.

Here are some excerpts.

7:31 But when it comes to high availability, really, it's tough.

And cloud isn't a silver bullet for availability.

7:58 Like everybody assumes that there's this big red button that says I've got high availability in any cloud. We're not just talking Azure, we're talking AWS and GCP.

And the cloud doesn't eliminate traditional availability issues.

8:54 You know, like your platform layer, meaning Azure or GCP, like I can spin up this VM in another region when something happens is great. But the problem then winds up being it doesn't account for things like transactions in a SQL Server installation. So all the traditional things we worry about on premises still exist up in the cloud with IaaS. It's PaaS where things get different.

And the cloud introduces new issues.

13:15 … but now we need to think about placement of our VMs. We need to think about like ensuring that the VMs don't sit on the same host or in the same rack, which means setting certain things differently in your cloud providers.

And old issues don't go away.

15:10 … I appreciate the idea that if you're responsible high availability, … that doesn't go away with the cloud. … You still are responsible for [high availability] and you have to work through the additional complexity.

And monitoring gets even more important.

23:11 And you really don't have a view of is it functioning well. And then you need somehow you need to be able to monitor the stuff remotely and realize we have no insight into what's going on into our systems. I saw an uptick in implementation of like a lot of the third party monitoring tools or SCOM … because people "Like we need this stuff." Whereas they didn't put too much importance on that.

But cloud providers likely provide higher availability than you could.

28:15 I still think the cloud is up more than any data center I ever ran was. What I've actually told customers is Microsoft, Amazon, and Google can outspend you in building a data center. And a 24 hour staffed NOC is normal for them and expensive for anybody else.

But just moving to the cloud doesn't guarantee more benefits.

30:46 So I can take my on premises VMware infrastructure, vMotion it up to Azure, AWS, GCP and vice versa all day long, right. … I think for some folks that's a good solution. … It gets you to the cloud. It works well sort of. … And so it's always interesting to see things like this is where it's more of a on premises play in the cloud.

If you're not good at on premises, you won't be any better on the cloud.

34:56 If you sucked at virtualization on premises, you're generally going to suck at stuff in the cloud. Not always, but it's generally true because if you couldn't size and you couldn't deploy well on premises, how are you going to automate and do all this cool dev ops, dev stack ops stuff. If you had no skills to do it on premises, it's not magically going to happen in the cloud.

AWS - Amazon Web Services

GCP - Google Cloud Platform

IaaS - Infrastructure as a Service

PaaS - Platform as a Service

SCOM - Microsoft System Center Operations Manager

NOC - Network Operations Center

Cloud Management Skills

Recently McAfee published a report on cloud adoption and risk. The Register did a review of the McAfee study.

The Register concluded:

The ongoing rash of data leaks caused by misconfigured clouds is the result of companies having virtually no visibility into how their cloud instances are configured, and very little ability to audit and manage them.

That's really scary. But it supports my concerns that moving workload to the cloud doesn't eliminate work/effort, i.e. manpower, but rather changes the skills required.

An organization should thoroughly understand the skills change that moving to the cloud brings.

These changes may be addressed with training but may require changing personnel to acquire the appropriate skills. These changes and the change management time and effort should be incorporated in the project timeline and budget for cloud implementation.

McAfee observed:

It’s possible the speed of cloud adoption is putting some practitioners behind.

The number 1 cause of cloud security issues noted by McAfee was "Lack of staff with the skills to secure cloud infrastructure."

That entire list is a good reference when considering moving workload to the cloud.

Cloud Security

I follow a number of CIO-orientated blogs and this article popped up recently:

Why promises about cloud security make me uneasy

I couldn't have said it better myself.

The entire article is worth reading but I'll try to pull out the highlights here.

Cloud companies will tell you that their products and services are secure and you can use security products to ensure that your data and applications are secure. I am a bit skeptical.

I can maintain and monitor use and access to my user base and interrogate logs of application use however I lack key visibility when in comes to the cloud...

While many organizations struggle to "maintain and monitor use and access" at least they control their own destiny. In a cloud environment you will "lack key visibility."

You'll recall my observations in a prior post.

...as a CIO are you comfortable with this level of opacity to service failures?

From the referenced article:

I lack control of the cloud and am dependent on a third party vendor for disclosure and transparency ...

Not a comfortable position for a CIO.

To raise both sides of this topic, if you or your organization are not able to "maintain and monitor use and access" maybe delegating that to a cloud provider is the prudent thing to do.

Just make sure your boss is on board with your decision. And that you've got the mobile number of your cloud salesperson.

Rumbling Clouds

I've written several times about "cloud" availability/reliability. A recent article on Microsoft's outlook.com service brought this back to my attention. This is Microsoft's free service so there's not really a lot of room to complain. This discussion is to compare the cloud solution to a self-hosted solution.

Microsoft has a status page which was getting updated regularly which is good but the details were a little light, e.g. "focused on remediation" and "users may be able to access the service by logging in to their Outlook.com accounts with a web browser as an alternative method". The problem existed almost a week.

Again this is a free service but these are situations you need to consider as you explore moving a service to the cloud.

Go read some of the comments:

When you go cloud hosted you are subcontracting your responsibility to a third party.

That third party may be much more capable than your IT budget will allow

I've got better reliability and almost the same service and functions as my company OWA account, free of charge as well.

Cloud is a way to free up the expense of operating your own IT services.

But putting your eggs into the hands of Microsoft, Google, Amazon or any other single entity to which you represent 0.00000000001% of their annual income is a stupid idea.

Realize that you are less than a rounding error to the cloud provider.

I don't hear many people talk about ... the time it takes to repair a massive amount of infrastructure even when know how and have a fix available. I expect this to only get worse as the cloud continues to grow.

Clouds by nature are BIG. Big things take a long time to fix. Be prepared for that.

It's Not Our Fault

I follow the Internet Storm Center's diary. Recently one of the entries related a situation with a personal NAS (Network Attached Storage) with terabytes of data. It was configured with RAID5. The NAS vendor offered a cloud backup service that he used.

He had a detailed backup plan consisting of:

• a daily backup to a cloud storage provider
• a monthly backup to an external disk (physically stored away from the source)
• a file restore test performed every month (ex: restore file ‘x' backup at time ‘t’)

While this diary is about a personal situation the lessons apply to enterprise as well.

The power failed in his area and the NAS crashed hard. He reloaded the operating system and rebuild the RAID5.

No data was lost.

But then he had to “relink” the existing cloud backup with the new backup task on the NAS. That  failed with strange error messages saying that some files were not found.

After the normal "Turn it off and back on" with no success the vendor responded:

Thank you to try to log on your cloud service console to check if your files are available. If they are not available, please contact your cloud service support to get more help. We already notified them about this issue and we received a lot of complaints from other customers who are facing the same issue. You should try to see with them how to recover your files, if possible...

Remember that "your cloud service" was arranged by the NAS vendor.

Yep, his cloud backup was lost (1.5TB of data).

What if that had been YOUR enterprise data?

Have a plan. Have a backup to that plan. Test it. Test it again.

A Fool and His Data Are Soon Parted

Azure just keeps having problems. And a recent incident happened while their CEO was speaking at a conference in Sydney "focused on showing off how Microsoft's cloud technology can have a transformative impact on businesses."

While it's easy to pile on with Azure, the situation is endemic to the cloud environment. Not withstanding my recent post, cloud implementations still carry significant risks. The news.com.au article states:

While cloud technology can have a massive impact on the way businesses run, as we are currently witnessing, it can also create new headaches when things don’t work how they’re supposed to.

And this was the second Microsoft cloud outage in 24 hours.

Really worrisome to me is that the Microsoft Office portal wasn't responsive.

As usual the comments are the most interesting. Clearly some are trolls but some are really to the point.

captain goodvibes
Anyone who trusts cloud technology to store data risks it disappearing into thin air. The only way to safely store data is on a hard drive. To paraphrase an old adage, a fool and his data are soon parted.

Anthony
The problem is for a lot of organisations is that they are too small to safely and economically run their ICT. Large vendors can provide resilience, security, and responsive services easier. And no, I don’t work for Microsoft, but I have worked in outsourcing :-) so I know the numbers

HG
If your business model is a monopoly, make sure your cr*p works.

Gordon
I spent most of my career in IT, I just do not understand why anyone would trust another company to manage all of their business critical data off site....

Yeah, worry about your business critical data.

Cloud Single Points of Failure

Probably one of my most frequent blog topics is "cloud". I'm sure that you've all read and incorporated my suggestions in your solutions. You're welcome.

Recently, a twist arose that I hadn't considered.

You've certainly taken advantage of your cloud providers geographic separation. But did you ask the really tacky questions?

In July, Google Cloud had an "incident" that took down customer workloads that were spread across multiple geographic "zones."

It turns out that some of Google Cloud services are tied to a single datacenter.

But YOU certainly inquired about that when you contracted to place your workload across different zones and regions to ensure resilience. And a requirement that all of the cloud's services you require were redundant in your contract with service level commitments.

Didn't you?

Now you know.

The details are here (archive.org) and here (archive.org).

Storm Clouds

During the last week in June, several of Microsoft's cloud services had outages. This is not picking on Microsoft as Google had outages as well. It's just that I found a good discussion about the impact of the Microsoft outages.

---

In outages this week, Microsoft’s online Exchange service was down for nine hours, crippling Office 365 and hosted Outlook accounts across North America and Mexico, just after its unified communications service also crashed.

Aside from the fact that entire businesses were left unable to read or write email in the Office 365 downtime, users were also pretty ticked off with Microsoft’s response. User Trec posted on the Office 365 community page:

Tried to contact support by phone, and after half hour, the call was dropped on both occasions. It seems the support guys are saturated and there are not enough staff for an issue like this one.

While Jim1001 said:

Our entire corporation cannot send or receive emails from Outlook (Office 365 Exchange) or even the OWA web browser as of 8AM MST time this morning June 24, 2014! I have never seen a world-wide email go down like this.

And MacBuffalo said:

Office365 is beginning to look like a very poor choice for mission critical services.

Lync Online, which provides VoIP, corporate IM and video conferencing had also briefly crashed two days before, leaving some customers, like ThomasGallaway, suffering a double whammy:

Down for 3 hours now. When going to Calendar outlook freezes. Send/receive works on iPhone. This sucks as we were hit yesterday by the lync outage. Today email. What's going on M$?

---

Just as in previous cloud services outages the provider (just happened to be Microsoft in this case) couldn't handle responding to customers about the situation. This still makes me worry about putting a corporation's systems in a public cloud where the provider can't provide the same level of accountability as internal services.

Unlike previous cloud services outages Microsoft's Rajesh Jha, Corporate Vice President, Office 365 Engineering has offered an explanation of the outage. The Corporate Vice President of Customer Service has been quiet.

My previous posts on this topic:

     When Clouds Go Bump
     When Clouds Go Thump
     Lessons from the Cloud
     When Clouds Go Bump Revisited
     To Be Fair
     To Be Fair, Again
     To Be Fair, Again and Again

If You’re Going to Use the Cloud

... for Pete's sake, please use its strengths.

You know I have mixed opinions on the "cloud" depending on the size and capability of your organization.

An example of leveraging the cloud's strengths is in a recent article I saw from KnowBe4 entitled "Can the Microsoft 365 Platform Be Trusted to Stop Security Breaches?"

KnowBe4 referenced an article from Hornetsecurity entitled "1 of every 4 companies suffered at least one email security breach, Hornetsecurity survey finds." (Don't click on that just yet.)

Realizing that everybody has an agenda, let's look at these articles.

KnowBe4 calls out the following findings:

• 33% of organizations are not using Microsoft’s multi-factor authentication (MFA)
• Of those using MFA, 55% of organizations are not using Conditional Access which scrutinizes connection requests beyond just providing credentials and additional authentication factors
• Only 43% leverage Microsoft’s data loss prevention policies to keep data from leaving the organization
• 68% of organizations expect Microsoft to keep email safe from threats

This is my point. If you're going to use a cloud solution such as Microsoft 365, leverage its capabilities. Even if they are premium services, they're probably NOT services you could deliver yourself.

KnowBe4's recommendation: Have your "Users ... undergo continual Security Awareness Training."

By the way, that's KnowBe4's business model - training users. And that's a good thing.

Now, before you click on Hornetsecurity's link, get ready for a pretty aggressive privacy policy.

That almost scared me off. But just click on "Cookie-Details" and slide everything to "Off".

In addition to the points that KnowBe4 raised from Hornetsecurity's study, Hornetsecurity has one more finding: "An impressive 82% of all our respondents who use third-party email security solutions reported no breaches."

I'll bet you can guess what Hornetsecurity sells.

Regardless of the various agendas, the Hornetsecurity study is solid and the findings valuable. Take them into consideration for your Microsoft 365 implementation.

And consider the value-add capabilities of any cloud solution you implement.

Big Blue Clouds

I've recently added Ben Kepes to my reading list. The Forbes web site is awfully heavy with clutter but I let Adblock Plus do its thing.

A recent article was about IBM (hence "Big Blue" in the title) and their cloud activities.

Ben drug out all the old stories, e.g. Watson winning Jeopardy and "nobody ever got fired for buying IBM" but went on to relate a conversation that he had with Robert LeBlanc, SVP of IBM’s Cloud business.

LeBlanc's comments are full of platitudes - "No enterprise is going to lift trillions of dollars of existing assets into the cloud." and "New development will happen in the cloud, but existing assets will stay put."

Then LeBlanc got into classic IBM speak. Kepes asked him if IBM "would begin to zero rate international traffic." LeBlanc's response was that "high availability is important."

Huh?

Kepes did call out IBM on their continual claims to be the biggest cloud vendor. "I've not seen any independent commentator who gives their claims any real credence."

Enterprise Cloud Providers

As you know, I'm a big follower of cloud technology. Early this year I came across an article in the Wall Street Journal.

In this article the WSJ reported on a recent survey by Piper Jaffray of 112 CIOs. Piper Jaffray asked the CIOs to name their preferred public cloud provider.

Amazon Web Services led the responses with a 2 percentage point increase from 33% to 35%. Microsoft's Azure increased from 20% to 21%. IBM Softlayer had the biggest gain increasing from 4% to 6%.

The big loser was Google falling from 12% to 7%.

Reminds me of the old saying that "Nobody got fired for choosing IBM." Or Microsoft.

There's an interesting comparison of Amazon and Microsoft's cloud offerings here.

Apple Cloud Outage

Now we have a new player in the battle of (un)availability in the cloud. On March 11, 2015 Apple's iTunes store was down for 10 plus hours.

But look closer at that web page. First, we could see that web page unlike one of Microsoft's outages. Second, nothing was down but the App Store. iCloud Account & Sign In - Up. iCloud Backup - Up. iTunes in the Cloud - Up. And on and on.

And Apple acknowledged the same day that the downtime was caused by problems with its DNS setup.

The other cloud players could learn something from Apple.

Google Cloud Outages

I've written numerous times about cloud availability. Last year I posted about an outage in Microsoft's Azure service.

Some of my comments were:

Recently there was an 11 hour outage of Microsoft's Azure storage services.

Again users were hard pressed to get details on the outage as "the Service Health Dashboard and Azure Management Portal both rely on Azure."

Cloud outages don't have to be like that.

Here're the communications from Google on a pair of recent outages in their cloud services.

Gmail Outage

November 3, 2015 1:21:00 AM PST

We're investigating reports of an issue with Gmail. We will provide more information shortly.

November 3, 2015 1:38:00 AM PST

Our team is continuing to investigate this issue. We will provide an update by November 3, 2015 2:38:00 AM PST with more information about this problem. Thank you for your patience.

This issue is affecting IMAP and SMTP delivery

November 3, 2015 1:48:00 AM PST

Our team is continuing to investigate this issue. We will provide an update by November 3, 2015 2:48:00 AM PST with more information about this problem. Thank you for your patience.

This issue is affecting incoming POP, SMTP and IMAP connections.

November 3, 2015 2:25:00 AM PST

The problem with Gmail should be resolved. We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better.

Four posts in just over one hour.

Google Calendar Outage

November 4, 2015 8:20:00 AM PST

We're investigating reports of an issue with Google Calendar. We will provide more information shortly.

November 4, 2015 9:25:00 AM PST

Google Calendar service has already been restored for some users, and we expect a resolution for all users within the next 1 hours. Please note this time frame is an estimate and may change.

November 4, 2015 10:25:00 AM PST

The problem with Google Calendar should be resolved. We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better.

Three posts in just over two hours.

Everybody is going to have outages. But this is the way to handle them.

Cloud - Office 365

Sometimes it's just too easy to make fun of a company.

In May 2017 Microsoft had gathered their favorite developers in Redmond for the annual Build conference. The focus of that conference was Microsoft's offerings of Office and Azure.

Yeah, you guessed it. Office 365 went down in the middle of the conference as documented by the Register.

Outages happen to everybody.

What I thought interesting were the comments on the Register post:

Cloud just means somebody else's computer that you have NO control over, and can go down at any time leaving you helpless.
Herby

Cloud: A computing service that signifies you do not know where any outage is, who is responsible for it, who else but you or your organization accesses the data on it, but nobody wants to do without.
Jeroen Braamhaar

Having worked with Cloud products for near 10 years now, I can say that in general they are much more reliable and lower cost than an on prem equivalent. ... [Y]ou do get a much better infrastructure from a company that invests millions/billions and has SLA's around it all than the bit of tin you got from the reseller round the corner.
Babbit55

This situation is kinda like when I was on a conference call with my CIO and his boss the CFO. It wasn't a pleasant discussion. When the call was over my CIO turned to me and said "I don't ever want to have that discussion again."

Don't find yourself or your company in the situation that Microsoft was in. Consider and act on the points in those comments.

When Clouds Go Bump

There certainly has been a lot of discussion at the CIO level about Cloud Computing. A recent CIO I spoke with was looking at the cloud to help him address his challenges in staffing and running his data center. He and I had deep discussions about his in house alternatives to the cloud. Interestingly while we were having these discussions one of the biggest clouds went bump.

On February 29, 2012 Microsoft's Azure product had a global service failure. Microsoft's Azure Service Dashboard was overwhelmed and pretty much unavailable for hours on end. When it was accessible here is one of the updates approximately 8 hours into the outage:

The restoration steps to mitigate the issue are still underway. This incident impacts Access Control 2.0, Marketplace, Service Bus and the Access Control & Caching Portal in the same regions where Windows Azure Compute is impacted. As a result affected customers may experience a loss of application functionality. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

This is not a Microsoft bashing. In April, 2011 Amazon also suffered a broad failure.

What I want to highlight is that cloud providers are not immune from service failures. They are likely capable of providing more redundant and resilient services than many organizations can provide.

My question is as a CIO are you comfortable with this level of opacity to service failures? Are you willing to answer your users and executives with one-way information flow from a (sometimes available) web page? Would a 10 day service credit make your CEO happy?

Lessons from the Cloud

While researching the recent Microsoft Azure Storage outage I came across a presentation by Jason Chan of Netflix. While the topic was "Practical Cloud Security" there were a couple of slides that I thought applied to many organizations' situation.

We all remember how Netflix changed their business model several years ago going from mailing DVDs to streaming video online. This resulted in a 37 fold increase in API requests in one year.

Realistically most organizations don't experience that kind of growth but any organization that plans to grow should look for lessons learned.

Prior to the cloud migration Netflix was running their own data centers. Here's what they were doing in their data centers.

Sound familiar?

I especially like the "snowflake phenomenon" metaphor. That means that everything is a one-off. Nothing is reused.

Netflix's vision was to get out of the data center business and into the cloud.

We want to use clouds, not build them.

Their target patterns were much different.

Think about how this transition will reduce your costs and improve your delivery speed.

Maybe you can stream a movie in your spare time.

More Lessons from the Cloud

There seems to be a recurring theme in my posts about outages in cloud services. While not trying to beat that dead horse there are certainly some lessons to be learned here.

Recently there was an 11 hour outage of Microsoft's Azure storage services.

Again users were hard pressed to get details on the outage as "the Service Health Dashboard and Azure Management Portal both rely on Azure."

I commend Microsoft for owning up to the root problem quickly and succinctly.

"Unfortunately the issue was widespread, since the update was made across most regions in a short period of time due to operational error, instead of following the standard protocol of applying production changes in incremental batches."
     http://azure.microsoft.com/blog/2014/11/19/update-on-azure-storage-service-interruption/

One of the comments summed it up best:

Overdependence.

So much tied into itself that there is no dependency tree - it is a pure network - thus issuing bad changes take down the entire net.

It can be a spectacular update process - minimum to no outage... but only if the updates work.

It also shows a major vulnerability. That central update can take down the entire company if it gets penetrated.

jessepollard
20 November, 2014 12:46

So, lessons...

1. Diversify - Don't build your notification tool on top of what you're monitoring.
2. Manage change - Don't let operational error bite you in the a**. Your execution has to be perfect. Users are unforgiving.

These don't apply just to cloud solutions. They apply just as much to your internal solutions.

My previous posts on this topic:

Storm Clouds
When Clouds Go Bump
When Clouds Go Thump
Lessons from the Cloud
When Clouds Go Bump Revisited
To Be Fair
To Be Fair, Again
To Be Fair, Again and Again

Update: Microsoft has published a thorough analysis of the problem with the corrective action. Good job Microsoft!

Cloudy Day

To some I'm sure that March 1st felt like April 1st. Really, that couldn't be happening? Amazon's S3 (Simple Storage Service) went down in their Eastern Region (Ok, it just had "high error rates").

But there are a couple of lessons to be learned from this.

First, it seems nobody is listening to me.

Cloud services aren't magical (even Apple's). They rumble. They go bump.

Don't abdicate your responsibilities to the cloud provider. If you need high availability make sure that that is what your contract guarantees.

In the March 1st S3 outage either lots of customers didn't feel they needed high availability or they misunderstood what they contracted for.

Make sure you are not surprised like "Docker's Registry Hub, Trello, Travis CI, GitHub and GitLab, Quora, Medium, Signal, Slack, Imgur, Twitch.tv, Razer, heaps of publications that stored images and other media in S3, Adobe's cloud, Zendesk, Heroku, Coursera, Bitbucket, Autodesk's cloud, Twilio, Mailchimp, Citrix, Expedia, Flipboard, and Yahoo! Mail (which you probably shouldn't be using anyway)." (source)

At the same time don't over buy. One of my customers was considering migrating their on-premise servers to Azure. As part of their on-premise setup they had a specific backup system and service. When I investigated Azure's service commitments I found that Azure's committed backup and availability met my customers needs and the customer could discontinue their backup system and service.

By the way Amazon did a thorough post mortem on the outage.

Second (and more concerning since they should know better), even Amazon had highly visible services down.

Reminiscent of one of Microsoft's outages Amazon's own online public dashboard was down along with many of Amazon's customer facing services, e.g. Amazon Fire tablets.

From Amazon's post mortem "we have changed the SHD administration console to run across multiple AWS regions."

Amazon, hadn't you thought of this before? What else have you overlooked?

Saleforce's Circular Dependency

I follow cloud vendors' outages. Broadly, I believe that cloud vendors can deliver higher availability than most SMBs can do themselves. Enterprises are a different discussion.

But I always get a kick of looking at various cloud vendors post mortem reports (archive.is).

Recently Salesforce had a DNS outage. Like other vendors, e.g. Microsoft, the Salesforce outage even took down their status page!

And look at the spin they tried to put on it.

"We're not blaming one employee," said Chief Availability Officer Darryn Dieken.

And then they threw him under the bus.

"For whatever reason that we don't understand, the employee decided to do a global deployment," Dieken went on.

They don't understand?

But wait, there's more...

"In this case," he went on, "we found a circular dependency where the tool that we use to get into production had a dependency on the DNS servers being active."

 

If you're going to run a cloud service, you've GOT to design to avoid these kinds of problems.

But Can You Do Better Than This

If you've been reading this blog, you know that I worry about cloud availability.

However, most of the time the big cloud providers have more availability and redundancy capability than almost any enterprise can provide.

For an example, Microsoft recently had an outage of its Multifactor Authentication (MFA) for Azure and Office 365 users in North America,

There's a report on it here. And here's Microsoft's Root Cause Analysis.

Go read the Next Steps. Here's #1:

1. Fine-grained fault domain isolation work has been accelerated. This work builds on the previous fault domain isolation work which limited this incident to North American tenants. This includes:

• Additional physical partitioning within each Azure region.
• Logical partitioning between authentication types.
• Improved partitioning between service tiers.

Do you think your enterprise could add "Additional physical partitioning," "Logical partitioning between authentication types," and "Improved partitioning between service tiers?"

If you could, you must be in the Fortune 100. And that's just the first bullet of actions.

These outages from the cloud providers get a lot of publicity but unless you're a mega-scale enterprise any of them are way more capable than you'll ever hope to be.

Between a Rock and a Hard Place

That's where ALL Insteon users find themselves.

I've had a good ride with Insteon. Until April 15, 2022.

I'll let you read Stacey on IoT's coverage. And even SmartLabs' feeble and belated attempt at explaining.

Where that leaves me and 1000s of other Insteon users is that their in-house equipment is fine but crippled due to https://connect.insteon.com being down for the count.

So, where do I go from here? There are several organizations out there trying to help.

But let's think about those solutions. They all cost a couple of $100s. And when you're done, you still have the Insteon switches and plugs which you can no longer buy and the Insteon hub which you can no longer buy. And the solutions put a server in your house that you have to support and maintain.

That's the Rock.

Then there are the alternatives. These mostly consist of Wi-Fi switches and plugs but depend on a cloud service.

What do you do if/when the cloud service goes away like Insteon did?

That's the Hard Place.

My decision is "Less is More." I'm going with Wi-Fi switches and plugs with no hub and server in the house. At least there are less moving parts.

And I plan to use Amazon Echo to create the automation schedules and routines. That gives me one degree of separation from the Wi-Fi switch and plug vendor's cloud service.

The End of Hegemony

Recently one of my mentors tweeted a link to an article entitled "Building and dismantling the Windows advantage." Lots of statistics and a good read. Don't worry. I'm not going to use big words like pl8t0 did.

I have a couple of thoughts on this.

First, the transition from Windows to Apple happens in "thunks." For example when I moved my wife from a BlackBerry (because Google dropped BlackBerry support) to an iPhone I should have heard the "thunk." The next thing you know I'd sold my hacked TouchPad running Android and bought an iPad. Now the iPhone and the iPad are connected (and I use that term loosely) with the iCloud. The "thunk" happened easily at my house. "Thunking" in an enterprise won't be so easy.

Current technologies encourage homogeneous platforms. Even Mac and Linux users who interact with other businesses tend to converge using Microsoft Office file formats. Remember back to the migration from .doc to .docx. (You have done that haven't you?) Today making an enterprise "thunk" from PCs to Macs is a big transition.

A side issue on this is that enterprises like to multi-source all their purchases. This doesn't necessarily mean that they have 2 vendors' products sitting side by side but that they have 2 vendors at least bid on a product, in this example a desktop PC. They may award all purchases to a single vendor for a fixed time (several years) then rebid the product. Since there's only a single source for Macs that would be a problem and require some rethinking of the acquisition process.

Windows 8 will be a mover and a shaker in this arena. It could go either way. It could frustrate enterprises with the forced Metro interface or, if Windows Phone is adapted by the enterprise, complement the mobile environment. Too early to tell. Why don't you get me a Microsoft Surface for Windows 8 Pro and I'll give you some feedback?

Second, I don't think the long term play is full function desktop devices. The "cloud" is the end game. I was speaking with an executive with a leading IT vendor recently and she was saying that she didn't understand why the iPad was taking off in the enterprise. She couldn't "do" anything with it. I pulled out my Android phone and showed her my instance of Epistle. On the iDevices I use Nebulous Notes Lite. And on the desktop/laptop I use Wordpad. These apps leverage Dropbox to give me seamless editing of simple text documents wherever I am on whatever device I have at hand. It's an incredible experience. I understand that she was looking to be able to edit PowerPoints and Excel documents but that's just an issue of maturity in these applications. With the "cloud" it doesn't matter what your client is.

I understand the enterprise concern over "cloud" security but I forsee that being overcome by private clouds, e.g. SharePoint.

Remember the Chinese curse, "May you live in interesting times." We certainly are.

There Is No Cloud

I post fairly regularly about the "cloud." I have mixed opinions depending on the size and capability of your organization.

But recently I was following a story of Google changing their desktop Drive solution, AGAIN.

As frustrating as that will be to Google Drive users, that's not the story I want to tell.

Often the comments on a tech story are as interesting and valuable as the story itself and this was no exception.

lglethal posted:

It's just another reminder that The Cloud is just someone else's computer. And if they decide to change the rules around using their computer, then you either have to suck it up and accept it, or to try and pull back in all your data yourself. Neither of which is usually an easy or pleasant experience...

Good advice.

Turns out, there's even a t-shirt for this!

Get yours now!

Chromebook

On the day the new Samsung Chromebook was announced, I pre-ordered one from BestBuy. Long story why but it arrived last Thursday.

My wife has used it as much as I have and she loves it. She likes it because it is light and cool. She normally uses a ThinkPad T61.

Me, I'm a little more objective.

Since it only has a 16GB SSD I ordered a 32GB SD card to supplement it. The SD slot in my T420 lets the card go completely inside the chassis so I just assumed that's what the Chromebook would do also. Wrong.

The picture on the left is of the empty slot and on the right the card is inserted. You can see that the card protrudes about 3/8". You can't practically leave it in there as you'll break it off. Grumble, grumble.

Also notice how the screen hinge sticks up noticeably. I know it only costs $249 but that's UGLY.

The browser on Chrome OS isn't the same as the Chrome Browser on Windows.

For example, I use the Chrome Toolbox. Here's the Chrome Toolbox v1.0.32 Options screen on Chrome 22.0.1229.94 m on Windows 7.

And here's the Chrome Toolbox v1.0.32 Options screen on Chrome OS 23.0.1271.49.

"Double click on a tab to close it" is why I use this extension. No, that's not an option on the Chromebook.

I've just begun to play with Chrome Remote Desktop. On Chrome on Windows it is v1.9.1271.42, On the Chromebook it is v1.8.1229.66 and still labeled "BETA".

On Chrome on Windows the Chrome Remote Desktop is an icon on the new tab screen.

Not so on the Chromebook.

So how do you invoke Chrome Remote Desktop? Chrome Remote Desktop is the last icon in the app list.

From there on, it works fine.

Stay tuned for more discussion of Chrome Remote Desktop.

Now a Chromebook lives and dies by the Cloud. So you'd expect the interaction with cloud services would be exemplary? Don't get your hopes up.

Ctrl+M (more on that keystroke combo in a little) brings up the Chromebook's File Manager.

Yeah, that's not what I was expecting either. Folders are just intermingled with files.

Just to compare, here's the web presentation of Google Drive (from the Chromebook).

Ok, now to keystroke combos. I was using Google Plus and wanted to go to the top of the page. So I just needed to hit Home. Sorry, no Home key.

Nothing in Help to point me to it. A Google search turned up Ctrl+Alt+Up Arrow. Give me a break! It takes 3 keystrokes to enter something as fundamental to web browsing as Home?

So you want to know what the rest of the secret codes are? Good luck. Try Ctrl+Alt+/.

I'm not kidding. That's the documentation. Just poke around and discover what does what. Unacceptable.

I'm not sure about the Chromebook.

Update: More here.

To Be Fair, Again and Again

To be fair, Google's services (sorry but I couldn't find another source) went down recently as did Microsoft's services, again.

To be fair, this is what Microsoft's Dashboard looked like:

Manual actions to perform Swap Deployment operations on Cloud Services may error, which will then restrict Service Management functions. At this time we advise customers to delay any Swap Deployment operations.

To be fair, this is what I got when I went to Google's Dashboard:

To be fair, Microsoft's lasted 24+ hours (for one function) compared to Google's 20 minutes (for 0.02% of the Gmail user base).

To be fair, both probably had lingering effects.

To be fair, this is what life in the cloud is going to be like for the foreseeable future. But there is a pattern here. The Register called it a "Blue Sky of Death."

Follow-Up: Microsoft talks to The Register.

iPad vs. Data Center

Given that I call myself an IT architect, the headline caught my eye.

How the iPad ruined the lives of IT architects

The author describes how consumer technology has affected data center expectations.

In current times however, we’re being asked to regularly provide levels of solution availability that until recently were reserved for the largest of enterprises ...

This comes on continuing escalation of complexity in our solutions. I recall a discussion with the CFO of a Fortune 100 company about her expectations of IT. I related to her the increased complexity of an SAP TCP/IP solution compared to the previous CICS SNA solution. She was understanding of the challenges but that's where we had to go. And not just with the same availability but better.

What has caused this elevation of expectations?

Today ... the consumerisation of high quality IT has happened and is setting the standard for business IT. ... As a result of this turnaround, the role of an IT architect has got even harder, especially in the small- and mid-enterprise sectors where arguably the pace of IT change has never been faster and the lack of IT governance has never been lower.

I think the last point is also key. In my recent experience in medium enterprises (sub $1B) I've found that the lack of governance is a major problem.

Back to technology...

With their iPads always working and Facebook always being online, business users increasingly have the same expectation of the IT systems they use.

One of the business requirements I recently encountered was a CEO who wants to be able to walk into a customer's conference room and project that customer's business real-time. Stop for a minute and think of all the IT capabilities required to do that. Yet Facebook does it all the time.

Helping business users understand, justify, and quantify their requirements is the skill of a good architect, and is a process we can still use to define availability needs even if it’s to show that ultra-high availability needs bring ultra-high costs.

To the aforementioned CEO, it's just the cost of doing business. The challenge for the architect is "selling" that requirement through the organization.

(T)he problem architects now have is the delivery of infrastructures to support these expected levels of 24/7 availability. Quoting 99.5per cent availability SLAs these days suggests to me that we want the business to feel grateful for whenever the solution is available.

The CEO won't "feel grateful for whenever the solution is available."

(T)he answer ... seems to be for everyone but the richest organisations will almost certainly be the cloud. That brings a new and bigger challenge for the IT architect, how do we learn to trust a face-less cloud service provider?

This is why I think of myself a a Chief Worrier.

The Cloud is Full

Recently ZDNet's  All About Microsoft reported that Microsoft Azure customers had reported hitting virtual machine limits in U.S. East regions.

This is scary.

There were a small number of comments (7) that related that they had had the same experience. Here is a typical comment.

GetNrDone
Happened to me. I tried to deploy a new SQL database in eastus2 2 weeks ago only to be greeted by an error. Opened a tickets and was basically told there was nothing they could do. Escalated the issue with our TAM which also could not get approval for 1 database to deploy. I was asking for the smallest database they offer (s0) and was told we couldn't have it. No communication before, no warning emails, no blog posts, nothing in the service dashboards, even our account team didn't know anything about it. Completely blindsided me and delayed development on an app for a week while i could move resources to another region. Unacceptable and definitely does not live up to promises made!

There's a reddit thread on this here.

dops0
We've faced this issue in North Europe, East US and West US 2. This has just started happening over the last couple of weeks and what's even more frustrating is, we already have sufficient quota allocated to us, but, our users haven't been able to deploy their machines even when within this quota.

This issue hasn't blown up so either it has been resolved or customers were able to work around it as GetNrDone did.

Regardless this is another consideration for using anyone's cloud services. I'd even suggest trying to putting a clause in your contract that guaranteed x% of available capacity on the vendor's part. I'd bet that none of the vendors would accept that but that would at least make the vendor play their cards.

Living In the Cloud

A while back I mentioned a couple of Cloud apps that I use.

Although I'm a huge Google user, I've been using Dropbox for even longer than Google Docs/Drive. And I've found that the app ecosystem around Dropbox is a lot more mature than around Google Drive.

I started off just trying to find way to maintain a To Do list that I could access from anywhere. I came across Epistle, an Android app, that lets you edit a text file from Dropbox on your Android device.

Here are my Epistle settings:

Obviously then you can edit the text files from your PC using Notepad or Writepad.

So my issue with To Do list was done. That worked so well I began looking for how how I could leverage that solution.

My wife and I are always making grocery lists on scraps of paper and then leaving them at home when we drive by the grocery store.

Epistle would fix that!

But my wife uses an iPhone.

One thing I don't like about iOS apps is that there are so many fewer free apps. It's not that I mind paying for apps, it's just that simple function should be simple cost, maybe even free.

After looking at a number of iOS apps that allow editing of Dropbox text files, I settled on Nebulous Notes Lite. As you'd guess from the "Lite" moniker, there's a paid version that eliminates the ads but it costs $5. Way too much.

Here are my Nebulous Notes settings:

My wife loves it. She'll update the grocery list and text me to stop by the grocery store and I've got a complete list.

So we're done.

Not so fast.

I got a new Google Chromebook. No Notepad there. Sure I could download a file from Dropbox, edit it using Google Docs, and reupload it to Dropbox but that wasn't cool.

So I Googled it some and found Writebox. It does the same thing as Epistle and Nebulous Notes but from a browser. Perfect for the Chromebook. And there's even a Chrome app if you want it more integrated.

Writebox doesn't have an automatic sync option like Epistle and Nebulous Notes so you have to press Ctrl+S when you're done.

The future is here.