RAID Problems

RAID is good. Right? Keep on reading.

First read this article in The Register.

tl;dr - "Rebuild times are so long that the chances of an unrecoverable read error (URE) occurring are dangerously high."

Here's the long version.

RAID 5 uses a parity set to recover from a failed drive. The problem is that spinning disks are getting larger. This increases the chance of an unrecoverable read error occurring during the rebuild of a failed drive from the parity set. Your data is gone at that point.

The math behind this is really complicated but here's the punch line:

Consumer magnetic disk error rate is ... an error every 12.5TB.

Now let's look at that for today's big drives.

Putting this into rather brutal context, consider the data sheet for the 8TB Archive Drive from Seagate. This has an error rate of 10^14 bits. That is one URE every 12.5TB. That means Seagate will not guarantee that you can fully read the entire drive twice before encountering a URE.

Gulp!

How big are the drives in your RAID? Mine are 2TB consumer class.

What's a person to do?

Buy more expensive drives.

Enterprise magnetic disk error rate is ... an error every 125TB.

That reduces the failure rate by an order of magnitude.

The elapsed time of the rebuild is still problematic. Realizing that a Drobo (1st generation) is not an enterprise class RAID system my experience is that the Drobo rebuild time is in excess of 24 hours per TB.

Or buy SSDs.

Enterprise SSD error rates are ... an error every 12.5PB.

That gets you another order of magnitude. But those are expensive.

There are alternative RAID modes that give more protection and better recovery time as well.

Big Blue Clouds

I've recently added Ben Kepes to my reading list. The Forbes web site is awfully heavy with clutter but I let Adblock Plus do its thing.

A recent article was about IBM (hence "Big Blue" in the title) and their cloud activities.

Ben drug out all the old stories, e.g. Watson winning Jeopardy and "nobody ever got fired for buying IBM" but went on to relate a conversation that he had with Robert LeBlanc, SVP of IBM’s Cloud business.

LeBlanc's comments are full of platitudes - "No enterprise is going to lift trillions of dollars of existing assets into the cloud." and "New development will happen in the cloud, but existing assets will stay put."

Then LeBlanc got into classic IBM speak. Kepes asked him if IBM "would begin to zero rate international traffic." LeBlanc's response was that "high availability is important."

Huh?

Kepes did call out IBM on their continual claims to be the biggest cloud vendor. "I've not seen any independent commentator who gives their claims any real credence."

An Order of Magnitude Improvement in Compute Density

Remember 2 1/2 years ago I posted on the emergence of ARM servers. It's taken a while but they're being adopted by mainstream enterprises.

During Applied Micro's 4Q conference call CEO and president Paramesh Gop said:

PayPal achieved "... an order of magnitude improvement in compute density"

That's the kind of leap you have to have to displace x86. This reminds me of the transition from mainframes to x86. But remember that RISC systems made a brief appearance during that time. ARM may not be the next big thing.

It's going to be interesting over the next couple of years.

Walled Gardens

At lunch with a friend recently we were discussing messaging directions. While his perspective was for a Fortune 100 corporation and mine is for our much smaller consulting organization, we were running into the same issues.

The players in the messaging sector are creating walled gardens to tie you into their ecosystem. I'm sure that they would argue that it is to "improve the customer experience" and it may very well do that for the in-house customers but isolates them from the rest of the world.

Here are a couple of examples that I've run into recently in my space:

Facebook ends support for AIM chat integration 

On April 15th 2015, Facebook is making an update to their API (application program interface) that affects how AIM and other applications connect to Facebook. 

As part of the API update, Facebook will remove support for third party clients like AIM to integrate with their chat feature. You will continue to receive Facebook notifications in your AIM Updates feed and everything else you’ve come to know and love about AIM will stay the same.

Google Talk was a player for a while.

Google moves away from the XMPP open-messaging standard 

Google has since admitted that it is indeed shrinking its support for Extensible Messaging and Presence Protocol (XMPP). 

XMPP was meant to enable users from one Internet communication network to be able to talk to a friend or co-worker on another such network. So, for example, an AOL Instant Messenger (AIM) user could talk to his co-worker on Google Talk and vice-versa. 

May 28, 2013 

I used to use imo.im but they withdrew into their own walled garden.

Popular messaging app imo.im to end support for third-party IM networks starting Mar. 3

In emails sent out to users on Friday, imo announced that it will be discontinuing support for all third-party instant messaging networks to focus on a build-out of its own platform. 

February 28, 2014

What do I use now? Ironically I use the web interface of aim.com. It still supports Google Talk. And it'll send SMS.

I'm a little embarrassed.

Vestalink Confusion

I've posted before about my use of Vestalink. It always worked well for me. When Google and Obihai reached an agreement I moved back to pure Google Voice but left Vestalink in place for E911.

Initially the package from Vestalink was $39.99 for 2000 incoming/outcoming minutes/month. When it came up for renewal it was $79.99 for the same package. That made me step back and think about it.

Coincidentally as the renewal approached I had posted about Your Father's Phone Service triggered by the senior citizens on their home page. Within a month I got this comment:

Ryan Tilton said...
Hi Ben:
We are actually doing landing page testing. Since fatherly individuals might be looking for a land-line replacement we were thinking this picture might lead to good conversions. We still do not have enough data to see if this is a good picture or not. But it is good to hear your feedback on it. It would be great if you could start a discussion on the validity of the official obihai support. I think there is some interesting things going on a Google right now with Nova (the new cell phone service). I believe that the "official" support Obihai is claiming is just to sell more units before Google voice is finally merged into hangouts later this year. VoIP is not free like email. Google is taking huge losses to support this network, they have to monetize it eventually. I think Nova is the ultimate result. Great blog by the way!

Ryan Tilton is the owner of Vestalink. He encouraged me to "start a discussion on the validity of the official obihai support" and suggested that the "'official' support Obihai is claiming is just to sell more units before Google voice is finally merged into hangouts later this year."

I did whine at him about increasing his rates.

..even the 500 minute plan is almost twice what I paid initially. Have your costs actually INCREASED this dramatically? Doesn't feel right.

He replied:

We want to significantly improve the software so we are raising prices to hire some programmers for system updates and better overall user experience.

And he lowered my rate for one year to the original $39.99. I still let the package expire.

Then recently I get an e-mail:

Vestalink Now Works With Google Voice

Uh, notice the picture! And it's still on the Vestalink homepage.

Then another e-mail:

Vestalink launches GVsip - GVsip allows you to use your Google Voice account with any VoIP device.

I'm getting dizzy.

Google's Toe

How did Google stump their toe so badly?

Read the fine print in this Google Apps Status Dashboard.

smtp.gmail.com is displaying an invalid certificate.

tl;dr Google let their smtp certificate expire on a holiday weekend.

Perhaps Google could put a reminder on their calendar to help them not to overlook this again. Better yet, put it on an Outlook calendar.

There's a good recap of it here.

RAVPower FileHub

We're planning a long trip this spring and I got to thinking about how to get media available for our granddaughter. Since I'm cheap frugal most of my iDevices are only 16GB and after iOS 8.1 there's not much room left for movies.

My first thought was to spread the media around, some on my iPad Air, some on my wife's iPhone 5S, some on my Moto X, etc. That was going to be confusing at best.

Then in my Deals RSS feed popped up a RAVPower FileHub. It's hard to explain what this gadget does. I'll start with sharing the description from Amazon.

RAVPower FileHub Built-in 6000mAh External Battery Pack & Charger with Wireless N Portable Pocket Travel Router, Wireless Micro SD TF Card Reader, Wireless USB, Wireless Flash, Mobile Storage Media Sharing, WLAN Hot Spot & NAS File Server

Whew!

It's got a connector for everything and it weighs 5.5 ounces. Its size is 2.9 x 2.8 x 0.9 inches.

I had been looking for a battery pack for my mobile devices anyway and my travel router is only 802.11g.

This seemed like it was too good to be true but it all seems to work.

I found an unused 32GB microSD card in my stash to contain the media.

It came already charged and when I turned it on it presented a Wi-Fi network named "FileHub-XXXX". You can connect to this with your PC with a password of "11111111". There's a web UI (10.10.10.254) that will let you update the firmware and configure the device.

There are iOS and Android apps in the respective app stores. Not only can you stream media from the microSD card to your Wi-Fi device(s up to 5) but the app has a player for media already on the phone or tablet. From your laptop browser just go to 10.10.10.254 and there's a media player served from the device.

If you give the app access to your device's media, you can download the media from the device to the FileHub using either the microSD or an external USB drive. Similarly if you access this from your laptop browser you can download files from the laptop to the FileHub using either the microSD or an external USB drive basically making it a microSD/USB adapter.

All this worked perfectly and I haven't found anything that it won't play although pretty much all my media is H.264.

I flashed the firmware to the latest rev.

I have used it as a Wi-Fi WISP device bridging my Wi-Fi to another SSID with NAT in the middle. Just what you need for Starbucks. It will also function as a pure NAT router generating its own SSID from Ethernet input.

The charging output is 1A so it probably won't do well charging an iPad.

Awesome!

Cloud Reliability Showdown

I keep following enterprise cloud technology and services. Ben Kepes is one of the bloggers I follow. He had a recent article analyzing a report from CloudEndure that compares the reliability of Amazon's AWS.to Microsoft's Azure in 2014.

The article is full of graphs and good analysis but his summary hits the nail on the head:

Outages happen - there is no way of avoiding mentioning that fact. But as Netflix has so successfully demonstrated, it is through planning for failure that organizations can achieve the very highest levels of reliability for their applications. Thinking about multiple points of redundancy for every service, every connection and every vendor an application uses is a good start.

See my article on Netflix.

And the he puts a point on it:

Both services are great and both show amazing reliability (as an aside, that reliability is far higher than most on-premises infrastructure) - the key takeaway from this report is that not planning for failure is a huge risk for organizations.

Put that in your pipe and smoke it.

Google Apps vs Office

Business Insider recently had an article on how Google plans to "nab 80% of Microsoft's Office business."

Google's plan is:

1. Make sure that the apps Google offers have "85-90% of the functionality" of Office.
2. Don't worry about the remaining 10-15% of the features required by power users, particularly Excel.
3. Support Office documents as a "first-class citizen."
4. Don't try and convince enterprises to convert from Microsoft Office to Google Apps.
5. Teach them to become power users.
6. Get new customers hooked on products other than Apps.
7. Show them how Google's cloud helps mobile workers.

Yeah, right.

Although I'm a huge proponent and user of Google's office apps for my personal use, there is no alternative to Microsoft Office documents in an enterprise environment. Segregating users into "have's" and "have not's" is going to be extremely difficult.

Apple Cloud Outage

Now we have a new player in the battle of (un)availability in the cloud. On March 11, 2015 Apple's iTunes store was down for 10 plus hours.

But look closer at that web page. First, we could see that web page unlike one of Microsoft's outages. Second, nothing was down but the App Store. iCloud Account & Sign In - Up. iCloud Backup - Up. iTunes in the Cloud - Up. And on and on.

And Apple acknowledged the same day that the downtime was caused by problems with its DNS setup.

The other cloud players could learn something from Apple.

Internet Bandwidth - February 2015

This is an update of my Internet Bandwidth post of June 2014.

The stacked bars are Comcast cable bandwidth. The yellow line is AT&T wireless bandwidth (scale is on right axis). The units are KB so 100,000 represents 100GB. The blue is the difference in what Comcast reports versus what my router reports.

I upgraded my AT&T plan to Mobil Share Value with 10GB in January 2014. AT&T upgraded my Mobil Share Value from 10GB to 15GB for the same price in November 2014. 

I started using CrashPlan Central for backups in December 2014. With CrashPlan Central uploads I've bumped into Comcast's bandwidth cap of 300GB. CrashPlan has good controls to manage its bandwidth usage and the initial upload is complete.

Screen Size Fragmentation

I'm a big user of Pocket Casts and got to following Russell Ivanovic's (one of the developers) blog. He has a couple of interesting posts on screen sizes.

His position is that the various screen sizes aren't a big deal so long as there are relatively few different proportions. You can make different resolution objects that the operating system can select from and present appropriately.

Here's his chart of the various Android screen proportions.

Now who hasn't heard that varying screen sizes are more of a problem on Android than on iOS?

From his follow on blog post here's the equivalent chart for iOS.

Note that the iOS chart was done with guesses of the iPhone 6 screen sizes and the guessed size of the iPhone 6+ was incorrect.

You decide which would be easier to code for.

Drive Snapshot

I have been using two partial backup solutions: IBM's Tivoli Continuous Data Protection and CrashPlan. They are somewhat different. I use Tivoli on selected directories to 1) backup to a second internal drive and 2) replicate that backup to an encrypted external drive that I keep in my car. I use CrashPlan to backup those same selected directories across the Internet to an external drive at my mother's. Tivoli is a commercial product, $54 for a permanent license. I use the part of CrashPlan that is free. Tivoli and CrashPlan both maintain versions of files in their backups.

But I got to worrying that I still had a lot of low value data that was not backed up anywhere. My drives are internal 400GB and 1TB drives and a 2TB Drobo (first generation, USB 2.0 attached). So I bought a 5TB external drive to backup everything.

My son-in-law is using Carbonite. When he had an internal drive fail recently he did a Carbonite recovery over the Internet. It took several weeks and generated several calls from Comcast about bandwidth usage. Subsequently he is using PureSync (free for personal use) to synchronize his Drobo with his internal drive. This doesn't address the Internet recovery situation.

But I wanted something to create pretty much an image of my drives so I could not only restore specific files but restore an entire drive if I had a drive failure. And I didn't want it to go over the Internet, not just for security but for bandwidth and recovery time.

Here's a bunch of sources that I looked at along the way.

• http://www.wilderssecurity.com/forums/backup-imaging-disk-mgmt.97/
• https://www.raymond.cc/blog/10-commercial-disk-imaging-software-features-and-backuprestore-speed-comparison/2/
• http://windowssecrets.com/top-story/mastering-windows-8s-backuprestore-system/
• http://www.maximumpc.com/hard_drive_cloning_software_2014?page=0,0
• http://www.wilderssecurity.com/threads/drive-snap-for-drive-snapshot.368010/ (wrapper for Drive Snapshot)

For the moment I'm testing Drive Snapshot. It is a commercial product (€39) with a 30-day trial. Unlike PureSync it doesn't require an install on the source system so I loaded the executable on the 5TB external drive. It takes compressed image backups. It takes differentials rather than incremental backups. It also has the ability to mount the backup file as a virtual drive which you can then access using Windows Explorer. It will do its own encryption but I'm a big fan of TrueCrypt so I'm just backing up to an encrypted volume.

I told Drive Snapshot to back up all of my volumes including the Drobo. It has to run as an administrator. It uses Windows Volume Snapshot Service to let you continue to operate while it's running.

Drive	Usage	Backup Time	Verify Time	In Use	Compressed
C	Boot	26:29	22:55	74.547MB	30.425MB
D	Data	463:51	344:50	572.508MB	559.835MB
E	Backup	141:09	126:18	224.437MB	207.060MB
I	Drobo	1223:26	1058:16	1.723GB	1.474GB
R	Restore	46:30	7:33	12.461MB	11.499MB

The times are in minutes. The Drobo took almost a day each for backup and verify. I believe that this is amplified greatly by the USB 2.0 interface. The Data drive and the Drobo are heavily JPEG and MPEG so they don't compress very well.

This is the Drive Snapshot log.

16:42:08 Disks in backup:
16:42:08 HD1:1 -> S:\HD1-1.sna
16:42:08 c: -> S:\C-2015-01-26.SNA
16:42:08 d: -> S:\D-2015-01-26.SNA
16:42:08 e: -> S:\E-2015-01-26.SNA
16:42:08 i: -> S:\I-2015-01-26.SNA
16:42:08 r: -> S:\R-2015-01-26.SNA
16:42:08 Preparing for backup
16:42:09 More than one disk in Backup, using VSS
16:42:09 Starting Snapshot creation
16:42:09 Creating shadow set {39e244a3-b75c-437d-abd6-d733b0e47110}
16:45:14 Register shadow copy {d3fda85e-198d-4eb5-acf7-6d32041c6c68}
16:45:14 Register shadow copy {646d142d-abbb-4b8b-88d1-367cf44beb28}
16:45:14 Register shadow copy {6499449b-0677-4ef0-ba3b-e8963be52ab6}
16:45:14 Register shadow copy {0dc8b0ab-c3e9-46eb-b0b7-e696dbb129f7}
16:45:14 Register shadow copy {ad35a58a-58d9-4b31-b122-07aa5e6ba58c}
16:45:15 The Volume Snapshot was created successfully
16:45:31 Start backup of HD1:1 -> S:\HD1-1.sna
16:45:32 free space info: total 102.396KB, 76.696KB free, 25.700KB must be saved
16:45:32 HD1:1 -> S:\HD1-1.sna
16:45:32 HD1:1 25.700KB in use - stored in 9.380KB - 0:02 minutes
16:45:32 Success
16:45:32 Start verification of: S:\HD1-1.sna
16:45:33 Success!
16:45:33 Start VSS backup of c: -> S:\C-2015-01-26.SNA
16:45:34 free space info: total 204.699MB, 130.152MB free, 51.203MB must be saved
16:45:49 c: -> S:\C-2015-01-26.SNA
16:47:03 c: -> S:\C-2015-01-26.sn1
...
17:10:12 c: -> S:\C-2015-01-26.s19
17:11:24 c: -> S:\C-2015-01-26.s20
17:12:03 c: 74.547MB in use - stored in 30.425MB - 26:29 minutes
17:12:03 Success
17:12:03 Start verification of: S:\C-2015-01-26.SNA
17:34:58 Success!
17:36:07 Start VSS backup of d: -> S:\D-2015-01-26.SNA
17:36:14 free space info: total 734.922MB, 162.414MB free, 569.508MB must be saved
17:40:15 d: -> S:\D-2015-01-26.SNA
17:41:39 d: -> S:\D-2015-01-26.sn1
...
01:15:20 d: -> S:\D-2015-01-26.374
01:16:49 d: -> S:\D-2015-01-26.375
01:18:51 d: 572.508MB in use - stored in 559.835MB - 463:51 minutes
01:18:51 Success
01:18:51 Start verification of: S:\D-2015-01-26.SNA
07:03:41 Success!
07:05:51 Start VSS backup of e: -> S:\E-2015-01-26.SNA
07:05:57 free space info: total 381.551MB, 157.114MB free, 221.437MB must be saved
07:06:44 e: -> S:\E-2015-01-26.SNA
07:07:45 e: -> S:\E-2015-01-26.sn1
...
09:21:53 e: -> S:\E-2015-01-26.137
09:22:51 e: -> S:\E-2015-01-26.138
09:24:52 e: 224.437MB in use - stored in 207.060MB - 141:09 minutes
09:24:52 Success
09:24:52 Start verification of: S:\E-2015-01-26.SNA
11:31:10 Success!
11:31:43 Start VSS backup of i: -> S:\I-2015-01-26.SNA
11:33:31 free space info: total 8.191GB, 6.468GB free, 1.720GB must be saved
11:39:35 i: -> S:\I-2015-01-26.SNA
11:40:41 i: -> S:\I-2015-01-26.sn1
...
07:47:24 i: -> S:\I-2015-01-26.1012
07:48:28 i: -> S:\I-2015-01-26.1013
07:54:38 i: 1.723GB in use - stored in 1.474GB - 1223:26 minutes
07:54:38 Success
07:54:38 Start verification of: S:\I-2015-01-26.SNA
01:32:54 Success!
01:36:24 Start VSS backup of r: -> S:\R-2015-01-26.SNA
01:37:45 free space info: total 14.143MB, 1.681MB free, 12.141MB must be saved
01:37:54 r: -> S:\R-2015-01-26.SNA
02:03:26 r: -> S:\R-2015-01-26.sn1
..
02:17:28 r: -> S:\R-2015-01-26.sn6
02:18:28 r: -> S:\R-2015-01-26.sn7
02:19:37 r: 12.461MB in use - stored in 11.499MB - 46:30 minutes
02:19:37 Success
02:19:37 Start verification of: S:\R-2015-01-26.SNA
02:27:10 Success!
02:27:11 Deleting previously generated shadow copy
02:27:26 Unregister shadow copy {d3fda85e-198d-4eb5-acf7-6d32041c6c68}
02:27:35 Unregister shadow copy {646d142d-abbb-4b8b-88d1-367cf44beb28}
02:27:37 Unregister shadow copy {6499449b-0677-4ef0-ba3b-e8963be52ab6}
02:27:39 Unregister shadow copy {0dc8b0ab-c3e9-46eb-b0b7-e696dbb129f7}
02:27:40 Unregister shadow copy {ad35a58a-58d9-4b31-b122-07aa5e6ba58c}

The full backups of all my data used about 1/2 of the 5TB drive. My plan is to take monthly differential backups. I should be able to take several of them before I will need to do another full backup.

The direct impact on the system was minimal. I saw Drive Snapshot up around 500MB of RAM and nominal CPU. Most of the time it was using less resources than CrashPlan.

However during the backup the system was very sluggish. Physical Memory hovered just below 100% of 8GB. It never wouldn't respond but was just really slow. This wasn't a problem for me as no one actually sits in front of that system. It cleared up after the backups completed. I think this memory usage was caused by the VSSs.

Time will tell.

Update: Differential Backups

After a month I had Drive Snapshot run a differential backup.

Drive	Usage	Backup Time	Verify Time	In Use	Compressed
C	Boot	22:01	2:45	74.934MB	4.144MB
D	Data	227:38	11:31	624.13MB	74.207MB
E	Backup	67:46	4:34	227.724MB	7.774MB
I	Drobo	1289:42	297:58	1.788GB	419.130MB
R	Restore	8:13	0:00	12.461MB	1.700MB

Generally the backup times are consistent with the initial backup. However the verify times are greatly reduced in line with the amount of changed data. Similarly the smaller compressed sizes are representative of amount of changed data. As expected the big change was on the Drobo. I had cleaned up a lot of data on it but added a lot of new data.

This is the Drive Snapshot log.

09:01:49 Preparing for backup
09:02:30 More than one disk in Backup, using VSS
09:02:30 Starting Snapshot creation
09:02:30 Creating shadow set {ec62783d-15d6-47ec-bb8b-ec6030845c26}
09:11:06 Register shadow copy {3a1957d6-e0a8-4d42-b7d8-b603fcb050f2}
09:11:06 Register shadow copy {214b2e72-25e4-435a-99ac-7dac07088515}
09:11:06 Register shadow copy {fbc66051-e1f1-42ec-b439-56b134f94b83}
09:11:06 Register shadow copy {33214159-e83c-44be-99c2-5b94fad0c447}
09:11:06 Register shadow copy {44ab91de-7ade-4d0e-aaff-6c692aae236c}
09:11:06 The Volume Snapshot was created successfully
09:11:13 Start differential VSS backup of c: -> S:\C-2015-03-02.SNA
09:11:15 free space info: total 204.699MB, 129.765MB free, 50.946MB must be saved
09:11:16 c: -> S:\C-2015-03-02.SNA
09:27:20 c: -> S:\C-2015-03-02.sn1
09:30:47 c: -> S:\C-2015-03-02.sn2
09:33:10 c: 74.934MB in use - stored in 4.144MB - 22:01 minutes
09:33:10 Success
09:33:10 Start verification of: S:\C-2015-03-02.SNA
09:35:55 Success!
09:37:12 Start differential VSS backup of d: -> S:\D-2015-03-02.SNA
09:37:19 free space info: total 734.922MB, 110.786MB free, 621.136MB must be saved
09:37:19 d: -> S:\D-2015-03-02.SNA
11:14:53 d: -> S:\D-2015-03-02.sn1
...
13:23:03 d: -> S:\D-2015-03-02.s48
13:24:02 d: -> S:\D-2015-03-02.s49
13:24:49 d: 624.136MB in use - stored in 74.208MB - 227:38 minutes
13:24:49 Success
13:24:49 Start verification of: S:\D-2015-03-02.SNA
14:12:20 Success!
14:13:15 Start differential VSS backup of e: -> S:\E-2015-03-02.SNA
14:13:18 free space info: total 381.551MB, 153.827MB free, 224.724MB must be saved
14:13:18 e: -> S:\E-2015-03-02.SNA
14:49:48 e: -> S:\E-2015-03-02.sn1
...
14:58:58 e: -> S:\E-2015-03-02.sn4
15:11:24 e: -> S:\E-2015-03-02.sn5
15:20:37 e: 227.724MB in use - stored in 7.774MB - 67:46 minutes
15:20:37 Success
15:20:37 Start verification of: S:\E-2015-03-02.SNA
15:25:11 Success!
15:29:06 Start differential VSS backup of i: -> S:\I-2015-03-02.SNA
15:31:01 free space info: total 8.191GB, 6.403GB free, 1.785GB must be saved
15:31:05 i: -> S:\I-2015-03-02.SNA
05:38:31 i: -> S:\I-2015-03-02.sn1
...
12:55:47 i: -> S:\I-2015-03-02.280
12:57:49 i: -> S:\I-2015-03-02.281
12:58:41 i: 1.788GB in use - stored in 419.130MB - 1289:42 minutes
12:58:41 Success
12:58:41 Start verification of: S:\I-2015-03-02.SNA
17:46:39 Success!
17:47:06 Start differential VSS backup of r: -> S:\R-2015-03-02.SNA
17:47:08 free space info: total 14.143MB, 1.681MB free, 12.141MB must be saved
17:47:08 r: -> S:\R-2015-03-02.SNA
17:54:58 r: 12.461MB in use - stored in 1.700KB - 8:13 minutes
17:54:58 Success
17:54:58 Start verification of: S:\R-2015-03-02.SNA
17:54:58 Success!
17:54:58 Deleting previously generated shadow copy
17:55:00 Unregister shadow copy {3a1957d6-e0a8-4d42-b7d8-b603fcb050f2}
17:55:06 Unregister shadow copy {214b2e72-25e4-435a-99ac-7dac07088515}
17:55:07 Unregister shadow copy {fbc66051-e1f1-42ec-b439-56b134f94b83}
17:55:08 Unregister shadow copy {33214159-e83c-44be-99c2-5b94fad0c447}
17:55:09 Unregister shadow copy {44ab91de-7ade-4d0e-aaff-6c692aae236c}

The differential backups used about 500GB. Looks like I can keep a couple of months of differential backups at a time and only do a full backup annually or semi-annually.

I think Drive Snapshot is a keeper.

GPT Drobo

I've been a Drobo fan since 2007. My original Drobo is soldiering along just fine.

It has grown from 371GB protected to 1.8TB protected. But storage demands are growing by leaps and bounds.

I had setup the original Drobo as a MBR (Master Boot Record) volume so it was limited to 2TB. And the original Drobo only supports up to 2TB drives.

So I had a problem.

GPT (GUID Partition Table) addresses the volume size and is supported by Windows 7. GPT supports up to  9.4ZB which should last me a while.

With 4 2TB drives the original Drobo will give 5.44TB protected.

I inherited another original Drobo and started building it up with GPT and big drives.

Initially I provisioned it with 3 1TB drives giving the same 1.81TB as my old Drobo.

Then I copied the data from the old Drobo to the new Drobo. That took a couple of days as it uses USB 2.0.

When the copy completed I checked the size and number of files copied and they were fine.

I plugged the new Drobo into another system and renamed the volume to the same as the old Drobo. I shut down SERVER and switched Drobos. After the boot I had to reassign the drive letter and correct sharing and permissions.

All was fine until I rebooted SERVER. Then I got "INVALID PARTITION TABLE." Clearly it was related to the new Drobo so I rebooted and chose the boot option to select the boot device. I chose the boot drive and up it came.

It appears that the original Drobo misrepresents itself to the BIOS early on as a floppy drive. In the BIOS boot search I still had floppies as the first entry even though I don't have a floppy. I removed this entry and the boot problem went away.

What's next? More space. What else?

After a week or two I'm going to reclaim one of the 1TB drives from the old Drobo and put it in the new Drobo.

This will give me 2.72TB of protected space, an increase of 910GB from the old Drobo.

The second next step is to replace a pair of the 1TB drives with 2TB drives.

This will give me 3.63TB of protected space. That'll have to wait until the price of 2TB drives comes down.

Update: When I added the additional 1TB drive one of the older 1TB drives failed. The Drobo did it's magic and seamlessly rebuilt redundancy with the new 1TB drive. This drove me to accelerate the swap to 2TB drives. My goal now is to stay far enough ahead of my capacity requirement that the failure of a single drive won't cause me to lose redundancy.

More on Internet Bandwidth

I've discussed Internet bandwidth before. Here I go again.

I've got to lay a little groundwork. My son-in-law uses Carbonite for backup. It's rock solid and they like the ability to view their photos using the Carbonite iPhone app.

I've been using CrashPlan with my own offsite storage. Then mid-December 2014 CrashPlan offered a promotional price on their online backup service and I picked that up.

Here's what it did to my bandwidth.

That sudden drop on Christmas Eve wasn't a Christmas present. That's when I realized I was going to blow by the Comcast 300GB bandwidth cap. I managed it pretty well coming in at 297GB for the month of December.

I turned CrashPlan's bandwidth usage back up in January.

So that's one side of the issue.

The other side is restoring over the Internet. My son-in-law had a hard drive crash and initiated a restore using Carbonite. Here's what his bandwidth usage did.

That's what 1.5TB of Comcast bandwidth looks like. And yes he got a call from Comcast.

And it took more than a month to complete the restore.

Incidentally it looks like Carbonite is resending the restored data back to their servers effectively doubling the bandwidth usage.

And recently my granddaughter spent the night with us. She's addicted to a children's show carried by Netflix. Can you guess what time she woke up?

Almost 10GB in under 4 hours!

That's a day's worth of my Comcast cap.

Again a non-trivial upload bandwidth, almost 20%. Odd.

Remember my forecast in June 2014:

3. I can already see Comcast's bandwidth cap of 300GB looming in my future.

We are there.

Here's a post by one of my Facebook friends and my response.

I'll follow-up on my backup alternative in a later post.

Google and Gander

Remember the old saying "What's good for the goose is good for the gander." Here's a good example.

Google's Project Zero publicly discloses flaws 90 days after it reports them to vendors. On January 11, 2015, Google disclosed a Windows 8.1 vulnerability. The problem was that Microsoft had committed to Google to fix it on January 13. Even without the fix potential attackers would "need to have valid logon credentials and be able to log on locally to a targeted machine."

At the same time it was discovered that Google was no longer fixing problems in the AOSP Internet browser in Android 4.3 (Jelly Bean) released July 24, 2013. When a security researcher notified Google of problems in the browser in the fall of 2014 he was told "we generally do not develop the patches ourselves but do notify partners of the issue." This affects 60 per cent of Android's active user base.

Don't hold your breath on getting a fix from Verizon or AT&T.

Incidentally, Microsoft supported Windows XP (released in 2001) until 2014.

Shame on you Google.

"And now the rest of the story."

Google is between a rock and a hard place on patching the AOSP browser. Let's say that they did patch it. It then would be up to the various vendors to incorporate that level of Android into their proprietary additions/changes to Android and then push it out the 900 million devices. Realistically the vendors won't do that. They'd much rather sell you a new phone.

On the other hand, it doesn't seem like too much for Google to patch the AOSP code and lay the blame for not updating the devices off on the vendors.

Update: Google blinked.

Enterprise Cloud Providers

As you know, I'm a big follower of cloud technology. Early this year I came across an article in the Wall Street Journal.

In this article the WSJ reported on a recent survey by Piper Jaffray of 112 CIOs. Piper Jaffray asked the CIOs to name their preferred public cloud provider.

Amazon Web Services led the responses with a 2 percentage point increase from 33% to 35%. Microsoft's Azure increased from 20% to 21%. IBM Softlayer had the biggest gain increasing from 4% to 6%.

The big loser was Google falling from 12% to 7%.

Reminds me of the old saying that "Nobody got fired for choosing IBM." Or Microsoft.

There's an interesting comparison of Amazon and Microsoft's cloud offerings here.

Google Keep

My quest for the perfect grocery list continues. I've been using I Sync Tasks as described here. The author has been very responsive and the service updates the shared lists in about 60 seconds. It costs $1 per person per month.

In November Google updated it's Keep service with sharing. It syncs in real time for free. I lke the price.

There are good clients for the web and Android. iOS is notable by its absence.

Certainly Google will come out with an iOS client but in the mean time I started searching. I came across an article on GuidingTech. It referenced TurboNote for iOS and GoKeep for iOS.

When I tested TurboNote for iOS it would never complete logging in as others have noted in the reviews. It hasn't been updated since June 2014 so I'd avoid it.

GoKeep for iOS fared much better. The reviews that suggest that this is just the web presentation aren't completely correct. The body of the app is the web presentation but it is "wrapped." There are additional functions in the wrapping including an online/offline toggle and a slide-in bar on the left. This is where you can enable the zoom capability. Some reviews note that you aren't able to delete a note. This is because the note deletion function is in the drop-down of the "..." at the bottom of a note. As some other reviews mention this drop-down menu just blinks and goes away.

I found that it was easier to use the Google Keep web site on a non-iOS browser to set the various options and then use the iOS app.

GoKeep for iOS was updated late Novermber 2014.

To get a feel for GoKeep for iOS browse to http://keep.google.com in Safari. Incidentally the "..." drop-down doesn't work there either. If that looks good then install the free version. The paid version for $1.99 is actually a separate app so you'll want to uninstall the free version and then install the paid version.

There are still some downsides to Google Keep. I used the Google Tasks ability to have subordinate list items. Google Keep doesn't support this. I also used the "widget" capability of Google Tasks to present the list on the Google Mail and Google Calendar web pages. Hopefully Google will bring these capabilities to Google Keep in the future.

Software-Defined SANs

The headline caught my attention - Server SANs: Don't throw the baby out with the bathwater

The author does a really good job of discussing the advantages and disadvantages of leveraging server storage as a virtual Storage Area Network (SAN).

I'm sure that the bean counters will jump on this and that the server salesmen will push this because it will help them sell more kit.

But this is really a bad idea.

While the following example is regarding ancient technology, the math doesn't change.

IBM 3380 Standard DASD had issues with internal paths. The details are arcane and not directly applicable to this discussion but suffice it to say that as you increased the capacity of the 3380 sub-system, the performance degraded. Worse the performance was different to different devices. While the average degraded gracefully, the standard deviation was all over the place. Siebo Friesenborg and I had a spirited discussion of this at the IBM Washington Systems Center.

The result was that the performance was erratic and very hard to manage.

Mapping the local storage of a cluster of servers into a virtual SAN can't help but suffer the same characteristics.

Storage that is mapped locally on a server will undoubtedly perform better than storage that is mapped 2 servers away.

Complicate the locality issue with the all too frequent I/O bandwidth issues of blade servers and you've got a solution that will make the performance consultants happy.

Call me.

Your Father's Phone Service

Remember in my journey to VoIP I encounted OBiVoice (now Vestalink). I signed up for their service when Obihai and Google had their little spat. For the price it worked really well.

After Obihai and Google kissed and made up, I revisited Vestalink as I was reconfiguring my OBi100 back to Google Voice.

When I hit Vestalink's home page, here's what I got.

What's up with that? And why are they looking at a Blackberry?

Hard Drive Costs

Everybody knows how fast hard drive costs are dropping, don't they? But really, let's look at the figures.

Here's a chart from That Data Dude.

Impressive. But look closer. The "Avg. Cost per GB" scale is logarithmic!

And look at the knee between 1989 and 1994. The cost per gigabyte dropped 98%. As I recall that was in the throes of  IBM 3390s. The 3390 was announced November 14, 1989 at 946MB. The last entry in the family was the 3390 Model 9 announced May 20, 1993 at 8.51GB. Bigger by a factor of 9.

Looking down at current costs they are pennies per GB. Several organizations with deep pockets have started offering "free" storage.

Think of the opportunities this will offer. Think of the problems this will cause.

Can You Afford for Google to Fail

I guess that headline got you to read this! No, I don't see Google failing as an organization anytime soon. What is evident though is that Google is part of the "Fail Fast, Fail Often" culture.

Google, like many other Silicon Valley companies, tends to iterate through products rapidly. Google so much that it precipitated a meme around Google's Graveyard.

That may be good for Google but is it good for you?

Even though I'm a huge Google user, I've joined the hue and cry about this.

     C'mon Google
     Google Plus Photos NOT
     Google Docs Gadget
     Google Being a Pain

One of the podcasts I listen to regularly is Speaking in Tech. While this is principally enterprise focused, episode #138 covered a couple of Google consumer issues. At 3:47 Greg Knieriemen goes off on Android Lollipop. Then at 10:47 the discussion turns to Google in general.

Sarah Vela puts a point on it.

(Google) can afford to fail in places where their users invest and then (the users) get burned. It makes me a little weary.

And John Troyer adds:

It also makes me wonder what happens when they are running my cloud. ... And ...  what happens when they're driving my car.

Well said.

Lessons from Sony

There's been a big story lately on a hack of Sony Pictures. Terabytes of sensitive data were exfiltrated and posted publicly. There're several theories about the motivations behind this but I want to focus on the security practices. Let's be slow to throw rocks because this could be you.

I'm a big proponent of leveraging size to reduce cost. Sony, Sony Pictures' parent company, had consolidated security management into its global organization. At first that seems like a good idea.

But the result was that the global organization couldn't/didn't focus on local issues. The global team was failing to monitor 149 out of 869 of Sony Pictures systems in their scope. That's 17% of the systems unmonitored.

And the global organization's IT management was aware of this gap and didn't remedy it. Even with 17% of the systems being unmonitored, almost 200 security incidents were reported to the global organization between September 2013 and June 2014.

It is not known if the penetration leveraged any of these unmonitored systems but they certainly were vulnerable.

Lesson: Cost should not be a primary consideration in IT security decisions.

There were also several issues that emanated from the leaked data. In the data were hundreds of RSA SecurID tokens, Lotus Notes IDs, passwords, and certificates - many of them with the required passphrase. One of the certificates was a certificate Sony Pictures used to sign code. Its password was the filename.

Lesson: Lock up the family jewels.

One of the other firestorms has been the content of the leaked e-mails. Beside all the sensitive business discussions were some pretty damning dialogs concerning actors and actresses.

Lesson: Have a policy about what is allowed in e-mail and recurrency training on the necessity of this policy.

Finally, face up to the fact that your company will be hacked.

Articles that I used in this post:
     Gizmodo
     Forbes
     CSOonline
     TechTarget
     TechTarget

Hara-Kiri

A random article in my feedly recently caught my eye. The headline was "H-P Moves to Retain Corporate Customers Ahead of Breakup." Well surely they would do that. So what? But I read it anyway.

(This article is in the Wall Street Journal and behind a paywall but if you Google that headline and click on the link from there you'll see the entire article.)

Stop now and go read it yourself.

The Palo Alto, Calif., company said it would offer versions of two computer server lines under H-P's Integrity moniker - Superdome and NonStop - that will be powered by Intel Corp.'s Xeon chips...

Whoa!

These Integrity systems now use Intel’s Itanium (formerly known as Merced) chips. They are pretty much the only users of these chips. At one time even Microsoft supported the Itanium chip.

This was in the era of system vendors differentiating themselves with their own chip architecture. HP previously had PA-RISC. DEC had Vax and Alpha. Sun had SPARC. IBM had PowerPC.

But this isn't a history lesson.

This is the end of purpose built processors. The Intel x86 has won (for now).

The Itaniums ran HP-UX, a version of Unix.

H-P is encouraging customers to move to the Linux operating system...
Antonio Neri, SVP and GM of H-P's enterprise group

Read that again. "H-P is encouraging customers to move to the Linux operating system..."

And on Intel X86 processors.

H-P has just committed hara-kiri.

They have gone from having a differentiating processor and operating system to being just another vendor of Linux and X86.

Sure they can put some lipstick on it with lots of processors and lots of salesmen but it'll still be just another Linux and X86 system.

It's a sad day.

Update: I shared this with the Unix manager at a Fortune 100 company. His comments:

I guess my take is this……. At least they actually documented a direction….. been struggling to figure out where they were headed for a couple of years now.

We’ll still shut them down as fast as we can though, just another O/S and vendor to manage…..

"Just another O/S and vendor to manage." Sad.

Your Grandfather’s Technology

Recently one of my co-workers sent me an article on the mainframe. Now you gotta remember that I'm a mainframe guy. I was into mainframes when 256 kilobytes was a BIG system. I led organizations that bet the company on mainframes. And won. I love to write S/370 assembler!

The article got this right:

...the mainframe computer seems like your grandfather’s technology...

That's me. But...

This article is self serving. And, in my opinion, misleading.

Somehow CA Technologies, né Computer Associates, have gotten the Wired brand in the top right corner. But all of the references are CA Technologies.

A CA Technologies survey found "more than 75 percent of U.S. respondents and more than 80 percent of global respondents confirmed that the mainframe is a strategic or highly strategic part of their current and future IT plans."

But dig a little and you'll find that the survey respondents were "623 mainframe executives." Not exactly objective.

That many organizations are sticking with the mainframe doesn't surprise me. In my consulting role, I see over and over that organizations won't/can't invest enough to implement new open systems. I call these organizations "Dead Men Walking."

And CA Technologies got this right: "...some pundits say the mainframe can’t possibly remain relevant in the cloud computing environment, where vast amounts of computing power and storage are available for anyone to rent at relatively low cost."

And I don't think I'd have used this example: "mainframes are still the go-to technology platform when it comes to tackling big jobs, such as managing the database of a government agency..."

I can't figure where this statement comes from: "...mainframes are hard to beat when you’re trying to plan for unpredictable spikes in network usage." Unpredictable spikes are the nemesis of monolithic mainframe systems. Horizontally scalable systems are made for unpredictable loads.

While I don't disagree that mainframes will continue to be around for a while it won't be because they perform the job better. And the business risk of operating on a mainframe will continue until those businesses migrate or fail. Only time will tell which.

More Lessons from the Cloud

There seems to be a recurring theme in my posts about outages in cloud services. While not trying to beat that dead horse there are certainly some lessons to be learned here.

Recently there was an 11 hour outage of Microsoft's Azure storage services.

Again users were hard pressed to get details on the outage as "the Service Health Dashboard and Azure Management Portal both rely on Azure."

I commend Microsoft for owning up to the root problem quickly and succinctly.

"Unfortunately the issue was widespread, since the update was made across most regions in a short period of time due to operational error, instead of following the standard protocol of applying production changes in incremental batches."
     http://azure.microsoft.com/blog/2014/11/19/update-on-azure-storage-service-interruption/

One of the comments summed it up best:

Overdependence.

So much tied into itself that there is no dependency tree - it is a pure network - thus issuing bad changes take down the entire net.

It can be a spectacular update process - minimum to no outage... but only if the updates work.

It also shows a major vulnerability. That central update can take down the entire company if it gets penetrated.

jessepollard
20 November, 2014 12:46

So, lessons...

1. Diversify - Don't build your notification tool on top of what you're monitoring.
2. Manage change - Don't let operational error bite you in the a**. Your execution has to be perfect. Users are unforgiving.

These don't apply just to cloud solutions. They apply just as much to your internal solutions.

My previous posts on this topic:

Storm Clouds
When Clouds Go Bump
When Clouds Go Thump
Lessons from the Cloud
When Clouds Go Bump Revisited
To Be Fair
To Be Fair, Again
To Be Fair, Again and Again

Update: Microsoft has published a thorough analysis of the problem with the corrective action. Good job Microsoft!

Chrome Memory

Recently I noticed my laptop running slowly. Processor utilization was nil so I fired up Task Manager to see what the memory usage was.

OMG!

Chrome was using over 4GB of RAM just from the processes on the first screen.

Here's what Chrome's Task manager showed.

I was running Chrome 38.0.2125.111 m (64-bit).

That just ain't right.

CryptoWhatever

One of our clients recently was impacted by CryptoWall. It's nasty.

Security researchers at Proofpoint warn that a new variant of CryptoWall recently spread through malicious banner ads. Surfers ran a risk of being faced with ransomware purely by visiting one of the impacted sites, which included various properties in the Yahoo!, Match.com, and AOL domains, among others. 

http://www.theregister.co.uk/2014/10/23/cryptowall_malvertising_outbreak/

This comment makes a good point that the ad networks should be called out for their participation.

This "drive-by" risk is very difficult to protect against. I believe that an ad-blocker will help mitigate this risk. I use Adblock Plus.

Lifehacker had a good article on all the things you can do with Adblock Plus. I also found that Adblock Plus has a page that will let you extend it.

Here're my options:

Adblock Plus is not without controversy. Read about it here.

X-ACR

So you block third-party cookies?

CHECK

And you opt-in to Do Not Track?

CHECK

And you run Facebook in an incognito/private window?

CHECK

And you block ads with Adblock Plus?

CHECK

So you think your privacy is protected?

WRONG

AT&T and Verizon are selling you out on your mobile device.

Verizon and AT&T (are) rolling this out: they’re tagging their customers with unique codes that are visible to third parties

Oh, and they're charging the third parties for YOUR information.

You can see what they're doing by following this link on your mobile device.

Here's what my headers look like.

If you're on AT&T you can go here and opt-out. Good luck though. It didn't do anything for me.

A detailed explanation of how it works is here.

The Electronic Freedom Foundation has this to say:

ISPs are trusted connectors of users and they shouldn't be modifying our traffic on its way to the Internet...

Amen.

Home Videos

With a granddaughter, I shoot a lot of home videos and share them to the family.

I upload all my video directly to YouTube. Then I use the YouTube editor to edit and consolidate my videos into a single video. I don't do any transitions or captions but you can do that there too.

Then I use ClipConverter to export the edited video from YouTube to my PC's desktop. I always download in the highest resolution available. Sometimes it takes YouTube hours to present the HD version so be patient.

Next I take the downloaded file and UPLOAD it to Google+. In my case I put it in the same Google+ album with the photos from a given event so that I have a single repository of an event.

Backup, Restore, Test

Do you backup? I sure hope you do.

Do you test your restore? What? No, you say?

My daily driver ThinkPad X201 only came with a 160GB drive. I picked up a 1TB Toshiba for $55 and started looking for a way to clone to it.

Then I thought that I use Windows 7's Windows Backup to make a system image every month. So why not use one of these to "clone" to the new drive?

I ran my system image backup as usual to an external USB drive. Normally when the system image is complete I rename the WindowsImageBackup folder to append the date and system name.

To "test" my restore, I renamed the folder that I wanted to restore back to WindowsImageBackup.

Then I shutdown the X201 and swapped the 160GB drive for the 1TB drive. I plugged in a USB version of Windows 7 64-bit System Repair Disk. I didn't plug in my backup drive yet.

I booted from the System Repair Disk and chose "System Image Recovery." Then I plugged in my backup drive. I had to click on "Retry" and then it found the image I had just made.

I just clicked on through and the restore started. It ran for a couple of hours and automatically booted back into Windows.

Just for giggles I ran SpinRite on the new 1TB drive.

That was really easy and now I'm confident that my system image backups work.

The restore only created a 160GB partition so I used Windows 7's Disk Management to resize.

There's a good tutorial on this process here.

Let's Hangout

Google's Hangouts (Chrome for Windows) is changing almost daily.

Google has been integrating SMS and Google Voice and now they've moved Hangouts to a floating window. The Chrome for Windows app also adds a Chrome App Launcher to the desktop and taskbar.

Not only does it launch Hangouts but also the other apps you have installed in Chrome.

Here's the list of what's new just in the latest version:

What’s new in the Hangouts Chrome app:
-- Native app experience. Hangouts runs in the background and notifies you of new messages and incoming calls with animated notifications. Launch Hangouts from the Start menu, have it pinned on your taskbar and switch apps using Alt+Tab.
-- Always-on-top avatars. Your friends’ avatars animate on your screen and stay on top so you never miss an important message.
-- Auto-minimizing chat windows. Chat windows auto minimize when you’re inactive to avoid cluttering your screen.
-- Message previews. Hover over any avatar to get a peek at the most recent message.
-- Drag and drop. Position the app anywhere on your screen, including multi-monitor configurations. You can also pop out important conversations and pin them to your desktop.
-- Keyboard Shortcuts. Navigate between chats quickly using your favorite Chrome shortcuts.
-- Google Voice support. Connect your Google Voice account to make calls, send and receive SMS, and access your voicemail.

Very interesting.

Auto Awesome

I still grumble about some features (or lack of features) of Google Plus Photos.

However, some of the features are awesome.

Recently we took our granddaughter to a nearby park. I took a sequence of photos of her while she was playing on the swing. Google Plus detected that these were framed similarly and automatically created an an animated gif of her swinging.

That's awesome but that ain't nothing.

The next thing that Google Plus did was that it determined that these same photos had a common background and erased her from the pictures and replaced her with the appropriate background.

I think Microsoft had something similar several years back that are mentioned here and here. If this was ever productized I can't find it.

The animated gif image in this post while an accurate representation of the Auto Awesome image was obfuscated with PhotoFiltre and recreated with Free Gif Maker.