Sunday, February 22, 2015

Drive Snapshot

I have been using two partial backup solutions: IBM's Tivoli Continuous Data Protection and CrashPlan. They are somewhat different. I use Tivoli on selected directories to 1) backup to a second internal drive and 2) replicate that backup to an encrypted external drive that I keep in my car. I use CrashPlan to backup those same selected directories across the Internet to an external drive at my mother's. Tivoli is a commercial product, $54 for a permanent license. I use the part of CrashPlan that is free. Tivoli and CrashPlan both maintain versions of files in their backups.

But I got to worrying that I still had a lot of low value data that was not backed up anywhere. My drives are internal 400GB and 1TB drives and a 2TB Drobo (first generation, USB 2.0 attached). So I bought a 5TB external drive to backup everything.

My son-in-law is using Carbonite. When he had an internal drive fail recently he did a Carbonite recovery over the Internet. It took several weeks and generated several calls from Comcast about bandwidth usage. Subsequently he is using PureSync (free for personal use) to synchronize his Drobo with his internal drive. This doesn't address the Internet recovery situation.

But I wanted something to create pretty much an image of my drives so I could not only restore specific files but restore an entire drive if I had a drive failure. And I didn't want it to go over the Internet, not just for security but for bandwidth and recovery time.

Here's a bunch of sources that I looked at along the way.
For the moment I'm testing Drive Snapshot. It is a commercial product (€39) with a 30-day trial. Unlike PureSync it doesn't require an install on the source system so I loaded the executable on the 5TB external drive. It takes compressed image backups. It takes differentials rather than incremental backups. It also has the ability to mount the backup file as a virtual drive which you can then access using Windows Explorer. It will do its own encryption but I'm a big fan of TrueCrypt so I'm just backing up to an encrypted volume.

I told Drive Snapshot to back up all of my volumes including the Drobo. It has to run as an administrator. It uses Windows Volume Snapshot Service to let you continue to operate while it's running.

Drive Usage Backup Time Verify Time In Use Compressed
C Boot 26:29 22:55 74.547MB 30.425MB
D Data 463:51 344:50 572.508MB 559.835MB
E Backup 141:09 126:18 224.437MB 207.060MB
I Drobo 1223:26 1058:16 1.723GB 1.474GB
R Restore 46:30 7:33 12.461MB 11.499MB

The times are in minutes. The Drobo took almost a day each for backup and verify. I believe that this is amplified greatly by the USB 2.0 interface. The Data drive and the Drobo are heavily JPEG and MPEG so they don't compress very well.

This is the Drive Snapshot log.

16:42:08 Disks in backup:
16:42:08 HD1:1 -> S:\HD1-1.sna
16:42:08 c: -> S:\C-2015-01-26.SNA
16:42:08 d: -> S:\D-2015-01-26.SNA
16:42:08 e: -> S:\E-2015-01-26.SNA
16:42:08 i: -> S:\I-2015-01-26.SNA
16:42:08 r: -> S:\R-2015-01-26.SNA
16:42:08 Preparing for backup
16:42:09 More than one disk in Backup, using VSS
16:42:09 Starting Snapshot creation
16:42:09 Creating shadow set {39e244a3-b75c-437d-abd6-d733b0e47110}
16:45:14 Register shadow copy {d3fda85e-198d-4eb5-acf7-6d32041c6c68}
16:45:14 Register shadow copy {646d142d-abbb-4b8b-88d1-367cf44beb28}
16:45:14 Register shadow copy {6499449b-0677-4ef0-ba3b-e8963be52ab6}
16:45:14 Register shadow copy {0dc8b0ab-c3e9-46eb-b0b7-e696dbb129f7}
16:45:14 Register shadow copy {ad35a58a-58d9-4b31-b122-07aa5e6ba58c}
16:45:15 The Volume Snapshot was created successfully
16:45:31 Start backup of HD1:1 -> S:\HD1-1.sna
16:45:32 free space info: total 102.396KB, 76.696KB free, 25.700KB must be saved
16:45:32 HD1:1 -> S:\HD1-1.sna
16:45:32 HD1:1 25.700KB in use - stored in 9.380KB - 0:02 minutes
16:45:32 Success
16:45:32 Start verification of: S:\HD1-1.sna
16:45:33 Success!
16:45:33 Start VSS backup of c: -> S:\C-2015-01-26.SNA
16:45:34 free space info: total 204.699MB, 130.152MB free, 51.203MB must be saved
16:45:49 c: -> S:\C-2015-01-26.SNA
16:47:03 c: -> S:\C-2015-01-26.sn1
...
17:10:12 c: -> S:\C-2015-01-26.s19
17:11:24 c: -> S:\C-2015-01-26.s20
17:12:03 c: 74.547MB in use - stored in 30.425MB - 26:29 minutes
17:12:03 Success
17:12:03 Start verification of: S:\C-2015-01-26.SNA
17:34:58 Success!
17:36:07 Start VSS backup of d: -> S:\D-2015-01-26.SNA
17:36:14 free space info: total 734.922MB, 162.414MB free, 569.508MB must be saved
17:40:15 d: -> S:\D-2015-01-26.SNA
17:41:39 d: -> S:\D-2015-01-26.sn1
...
01:15:20 d: -> S:\D-2015-01-26.374
01:16:49 d: -> S:\D-2015-01-26.375
01:18:51 d: 572.508MB in use - stored in 559.835MB - 463:51 minutes
01:18:51 Success
01:18:51 Start verification of: S:\D-2015-01-26.SNA
07:03:41 Success!
07:05:51 Start VSS backup of e: -> S:\E-2015-01-26.SNA
07:05:57 free space info: total 381.551MB, 157.114MB free, 221.437MB must be saved
07:06:44 e: -> S:\E-2015-01-26.SNA
07:07:45 e: -> S:\E-2015-01-26.sn1
...
09:21:53 e: -> S:\E-2015-01-26.137
09:22:51 e: -> S:\E-2015-01-26.138
09:24:52 e: 224.437MB in use - stored in 207.060MB - 141:09 minutes
09:24:52 Success
09:24:52 Start verification of: S:\E-2015-01-26.SNA
11:31:10 Success!
11:31:43 Start VSS backup of i: -> S:\I-2015-01-26.SNA
11:33:31 free space info: total 8.191GB, 6.468GB free, 1.720GB must be saved
11:39:35 i: -> S:\I-2015-01-26.SNA
11:40:41 i: -> S:\I-2015-01-26.sn1
...
07:47:24 i: -> S:\I-2015-01-26.1012
07:48:28 i: -> S:\I-2015-01-26.1013
07:54:38 i: 1.723GB in use - stored in 1.474GB - 1223:26 minutes
07:54:38 Success
07:54:38 Start verification of: S:\I-2015-01-26.SNA
01:32:54 Success!
01:36:24 Start VSS backup of r: -> S:\R-2015-01-26.SNA
01:37:45 free space info: total 14.143MB, 1.681MB free, 12.141MB must be saved
01:37:54 r: -> S:\R-2015-01-26.SNA
02:03:26 r: -> S:\R-2015-01-26.sn1
..
02:17:28 r: -> S:\R-2015-01-26.sn6
02:18:28 r: -> S:\R-2015-01-26.sn7
02:19:37 r: 12.461MB in use - stored in 11.499MB - 46:30 minutes
02:19:37 Success
02:19:37 Start verification of: S:\R-2015-01-26.SNA
02:27:10 Success!
02:27:11 Deleting previously generated shadow copy
02:27:26 Unregister shadow copy {d3fda85e-198d-4eb5-acf7-6d32041c6c68}
02:27:35 Unregister shadow copy {646d142d-abbb-4b8b-88d1-367cf44beb28}
02:27:37 Unregister shadow copy {6499449b-0677-4ef0-ba3b-e8963be52ab6}
02:27:39 Unregister shadow copy {0dc8b0ab-c3e9-46eb-b0b7-e696dbb129f7}
02:27:40 Unregister shadow copy {ad35a58a-58d9-4b31-b122-07aa5e6ba58c}

The full backups of all my data used about 1/2 of the 5TB drive. My plan is to take monthly differential backups. I should be able to take several of them before I will need to do another full backup.

The direct impact on the system was minimal. I saw Drive Snapshot up around 500MB of RAM and nominal CPU. Most of the time it was using less resources than CrashPlan.

However during the backup the system was very sluggish. Physical Memory hovered just below 100% of 8GB. It never wouldn't respond but was just really slow. This wasn't a problem for me as no one actually sits in front of that system. It cleared up after the backups completed. I think this memory usage was caused by the VSSs.

Time will tell.

Update: Differential Backups

After a month I had Drive Snapshot run a differential backup.

DriveUsageBackup TimeVerify TimeIn UseCompressed
CBoot22:012:4574.934MB4.144MB
DData227:3811:31624.13MB74.207MB
EBackup67:464:34227.724MB7.774MB
IDrobo1289:42297:581.788GB419.130MB
RRestore8:130:0012.461MB1.700MB

Generally the backup times are consistent with the initial backup. However the verify times are greatly reduced in line with the amount of changed data. Similarly the smaller compressed sizes are representative of amount of changed data. As expected the big change was on the Drobo. I had cleaned up a lot of data on it but added a lot of new data.

This is the Drive Snapshot log.

09:01:49 Preparing for backup
09:02:30 More than one disk in Backup, using VSS
09:02:30 Starting Snapshot creation
09:02:30 Creating shadow set {ec62783d-15d6-47ec-bb8b-ec6030845c26}
09:11:06 Register shadow copy {3a1957d6-e0a8-4d42-b7d8-b603fcb050f2}
09:11:06 Register shadow copy {214b2e72-25e4-435a-99ac-7dac07088515}
09:11:06 Register shadow copy {fbc66051-e1f1-42ec-b439-56b134f94b83}
09:11:06 Register shadow copy {33214159-e83c-44be-99c2-5b94fad0c447}
09:11:06 Register shadow copy {44ab91de-7ade-4d0e-aaff-6c692aae236c}
09:11:06 The Volume Snapshot was created successfully
09:11:13 Start differential VSS backup of c: -> S:\C-2015-03-02.SNA
09:11:15 free space info: total 204.699MB, 129.765MB free, 50.946MB must be saved
09:11:16 c: -> S:\C-2015-03-02.SNA
09:27:20 c: -> S:\C-2015-03-02.sn1
09:30:47 c: -> S:\C-2015-03-02.sn2
09:33:10 c: 74.934MB in use - stored in 4.144MB - 22:01 minutes
09:33:10 Success
09:33:10 Start verification of: S:\C-2015-03-02.SNA
09:35:55 Success!
09:37:12 Start differential VSS backup of d: -> S:\D-2015-03-02.SNA
09:37:19 free space info: total 734.922MB, 110.786MB free, 621.136MB must be saved
09:37:19 d: -> S:\D-2015-03-02.SNA
11:14:53 d: -> S:\D-2015-03-02.sn1
...
13:23:03 d: -> S:\D-2015-03-02.s48
13:24:02 d: -> S:\D-2015-03-02.s49
13:24:49 d: 624.136MB in use - stored in 74.208MB - 227:38 minutes
13:24:49 Success
13:24:49 Start verification of: S:\D-2015-03-02.SNA
14:12:20 Success!
14:13:15 Start differential VSS backup of e: -> S:\E-2015-03-02.SNA
14:13:18 free space info: total 381.551MB, 153.827MB free, 224.724MB must be saved
14:13:18 e: -> S:\E-2015-03-02.SNA
14:49:48 e: -> S:\E-2015-03-02.sn1
...
14:58:58 e: -> S:\E-2015-03-02.sn4
15:11:24 e: -> S:\E-2015-03-02.sn5
15:20:37 e: 227.724MB in use - stored in 7.774MB - 67:46 minutes
15:20:37 Success
15:20:37 Start verification of: S:\E-2015-03-02.SNA
15:25:11 Success!
15:29:06 Start differential VSS backup of i: -> S:\I-2015-03-02.SNA
15:31:01 free space info: total 8.191GB, 6.403GB free, 1.785GB must be saved
15:31:05 i: -> S:\I-2015-03-02.SNA
05:38:31 i: -> S:\I-2015-03-02.sn1
...
12:55:47 i: -> S:\I-2015-03-02.280
12:57:49 i: -> S:\I-2015-03-02.281
12:58:41 i: 1.788GB in use - stored in 419.130MB - 1289:42 minutes
12:58:41 Success
12:58:41 Start verification of: S:\I-2015-03-02.SNA
17:46:39 Success!
17:47:06 Start differential VSS backup of r: -> S:\R-2015-03-02.SNA
17:47:08 free space info: total 14.143MB, 1.681MB free, 12.141MB must be saved
17:47:08 r: -> S:\R-2015-03-02.SNA
17:54:58 r: 12.461MB in use - stored in 1.700KB - 8:13 minutes
17:54:58 Success
17:54:58 Start verification of: S:\R-2015-03-02.SNA
17:54:58 Success!
17:54:58 Deleting previously generated shadow copy
17:55:00 Unregister shadow copy {3a1957d6-e0a8-4d42-b7d8-b603fcb050f2}
17:55:06 Unregister shadow copy {214b2e72-25e4-435a-99ac-7dac07088515}
17:55:07 Unregister shadow copy {fbc66051-e1f1-42ec-b439-56b134f94b83}
17:55:08 Unregister shadow copy {33214159-e83c-44be-99c2-5b94fad0c447}
17:55:09 Unregister shadow copy {44ab91de-7ade-4d0e-aaff-6c692aae236c}

The differential backups used about 500GB. Looks like I can keep a couple of months of differential backups at a time and only do a full backup annually or semi-annually.

I think Drive Snapshot is a keeper.

Sunday, February 15, 2015

GPT Drobo

I've been a Drobo fan since 2007. My original Drobo is soldiering along just fine.

It has grown from 371GB protected to 1.8TB protected. But storage demands are growing by leaps and bounds.

I had setup the original Drobo as a MBR (Master Boot Record) volume so it was limited to 2TB. And the original Drobo only supports up to 2TB drives.

So I had a problem.

GPT (GUID Partition Table) addresses the volume size and is supported by Windows 7. GPT supports up to  9.4ZB which should last me a while.

With 4 2TB drives the original Drobo will give 5.44TB protected.

I inherited another original Drobo and started building it up with GPT and big drives.

Initially I provisioned it with 3 1TB drives giving the same 1.81TB as my old Drobo.


Then I copied the data from the old Drobo to the new Drobo. That took a couple of days as it uses USB 2.0.

When the copy completed I checked the size and number of files copied and they were fine.

I plugged the new Drobo into another system and renamed the volume to the same as the old Drobo. I shut down SERVER and switched Drobos. After the boot I had to reassign the drive letter and correct sharing and permissions.

All was fine until I rebooted SERVER. Then I got "INVALID PARTITION TABLE." Clearly it was related to the new Drobo so I rebooted and chose the boot option to select the boot device. I chose the boot drive and up it came.

It appears that the original Drobo misrepresents itself to the BIOS early on as a floppy drive. In the BIOS boot search I still had floppies as the first entry even though I don't have a floppy. I removed this entry and the boot problem went away.

What's next? More space. What else?

After a week or two I'm going to reclaim one of the 1TB drives from the old Drobo and put it in the new Drobo.


This will give me 2.72TB of protected space, an increase of 910GB from the old Drobo.

The second next step is to replace a pair of the 1TB drives with 2TB drives.


This will give me 3.63TB of protected space. That'll have to wait until the price of 2TB drives comes down.

Update: When I added the additional 1TB drive one of the older 1TB drives failed. The Drobo did it's magic and seamlessly rebuilt redundancy with the new 1TB drive. This drove me to accelerate the swap to 2TB drives. My goal now is to stay far enough ahead of my capacity requirement that the failure of a single drive won't cause me to lose redundancy.

Sunday, February 08, 2015

More on Internet Bandwidth

I've discussed Internet bandwidth before. Here I go again.

I've got to lay a little groundwork. My son-in-law uses Carbonite for backup. It's rock solid and they like the ability to view their photos using the Carbonite iPhone app.

I've been using CrashPlan with my own offsite storage. Then mid-December 2014 CrashPlan offered a promotional price on their online backup service and I picked that up.

Here's what it did to my bandwidth.


That sudden drop on Christmas Eve wasn't a Christmas present. That's when I realized I was going to blow by the Comcast 300GB bandwidth cap. I managed it pretty well coming in at 297GB for the month of December.

I turned CrashPlan's bandwidth usage back up in January.

So that's one side of the issue.

The other side is restoring over the Internet. My son-in-law had a hard drive crash and initiated a restore using Carbonite. Here's what his bandwidth usage did.


That's what 1.5TB of Comcast bandwidth looks like. And yes he got a call from Comcast.

And it took more than a month to complete the restore.

Incidentally it looks like Carbonite is resending the restored data back to their servers effectively doubling the bandwidth usage.

And recently my granddaughter spent the night with us. She's addicted to a children's show carried by Netflix. Can you guess what time she woke up?


Almost 10GB in under 4 hours!


That's a day's worth of my Comcast cap.

Again a non-trivial upload bandwidth, almost 20%. Odd.

Remember my forecast in June 2014:
3. I can already see Comcast's bandwidth cap of 300GB looming in my future.
We are there.

Here's a post by one of my Facebook friends and my response.


I'll follow-up on my backup alternative in a later post.

Sunday, February 01, 2015

Google and Gander

Remember the old saying "What's good for the goose is good for the gander." Here's a good example.

Google's Project Zero publicly discloses flaws 90 days after it reports them to vendors. On January 11, 2015, Google disclosed a Windows 8.1 vulnerability. The problem was that Microsoft had committed to Google to fix it on January 13. Even without the fix potential attackers would "need to have valid logon credentials and be able to log on locally to a targeted machine."

At the same time it was discovered that Google was no longer fixing problems in the AOSP Internet browser in Android 4.3 (Jelly Bean) released July 24, 2013. When a security researcher notified Google of problems in the browser in the fall of 2014 he was told "we generally do not develop the patches ourselves but do notify partners of the issue." This affects 60 per cent of Android's active user base.

Don't hold your breath on getting a fix from Verizon or AT&T.

Incidentally, Microsoft supported Windows XP (released in 2001) until 2014.

Shame on you Google.

"And now the rest of the story."

Google is between a rock and a hard place on patching the AOSP browser. Let's say that they did patch it. It then would be up to the various vendors to incorporate that level of Android into their proprietary additions/changes to Android and then push it out the 900 million devices. Realistically the vendors won't do that. They'd much rather sell you a new phone.

On the other hand, it doesn't seem like too much for Google to patch the AOSP code and lay the blame for not updating the devices off on the vendors.

Update: Google blinked.