I thought I’d update my series of posts on backup. The first one can be found here and the second part here.

Of course, there is no perfect backup strategy. Just like there’s no Santa Claus (sorry to dissapoint kids).

But depending on several things, like the volume of data you need to back up and the [...]]]>

I thought I’d update my series of posts on backup. The first one can be found here and the second part here.

Of course, there is no perfect backup strategy. Just like there’s no Santa Claus (sorry to dissapoint kids).

But depending on several things, like the volume of data you need to back up and the frequency you need to do it, it’s possible to design a backup strategy that comes pretty close…

After several months (or is it years) of researching, thinking, trying and failing and retrying, I think I’ve come up with a strategy that works really well. I’ll nut it out here so you can see what you think. For my needs and budget, I’ve decided to forego several technologies that many would see as essential or at least very desirable to solid backup. Namely, NAS (Network Attached Storage) & online backup. The advantages of this is that is provides a good basis for most people at an affordable level. This is clearly not Enterprise backup we are talking here. This is for home and small, small business users.

OK. The basis for any decent backup strategy is redundancy. A backup is not a backup unless it’s in at least 2 places (NOT including the original data).

Then there’s what to backup. I see backup split into two areas – data and system. Both need different approaches.

Finally there’s software. One for backup for data, one for the system drive.

For my data, I use SyncBackSE. I sync my data between my laptop and my desktop, and I back it up to an external 1GB drive, and a second external drive which I then keep off site (at my next door neighbours house). That gives me my data in 4 places, one of which is always off site and another which is off site sometimes (the laptop). I have the software run the backup profiles every night and I run them manually during the day if I’ve created something or have otherwise introduced new data. It’s an excellent bit of software and worth every cent ($25 or so). Small things like changing the Outlook data file’s location to the same directory as your data also make a lot of sense. All you data should preferably in one place.

Then there’s the system drive. I am trialling Acronis Tru Image, and it seems to be a very solid product. The idea here is that you can make a system image of the c drive and store it on (yet another) small external drive and for extra security a set of dvds. I also currently have my system drive in a RAID array. This is useful in the case of a total hard disk crash, but NOT in the case of an OS meltdown (any system problems get transferred to the RAID drive!) In future systems, I think I might forego the RAID Array and just rely on the mirroring technique. A great suggestion I got previously was to use a smaller, faster drive (80 or 100GB, but 10,000RPM) for the system drive to increase performance.

So that’s about it. I’d love to hear anyone’s ideas about it and by all means, let me know if you see any weaknesses in this strategy. I think it’s pretty solid.

In an earlier post, I spoke about back-up and the need to develop a system that does what it needs to do – save your data from all potential loss, and allow you to get back up and running in the quickest possible time.

Photographers have special requirements with regards backup. They generate huge volumes of [...]]]>

In an earlier post, I spoke about back-up and the need to develop a system that does what it needs to do – save your data from all potential loss, and allow you to get back up and running in the quickest possible time.

Photographers have special requirements with regards backup. They generate huge volumes of data in a very short time. It would be in no way unusual that in one photo shoot, I might generate from up to 4GB of data. And this is data, that if lost, is totally irreplaceable. The sheer size of it as well precludes online backup strategies for most, as upload speeds are too slow, and adsl account limits are often 20GB or less (this is of course only an issue if your ISP includes uploads as part of your allowance).

Peter Krogh, the DAM (Digital Asset Management) specialist, talks a whole lot about backup strategies in his book, The Dam Book. I’m not going into that much detail here, but there are a few things that are worthy of mention. There are many considerations when designing a backup system, including cost, scalability, and redundancy. With regards cost, he gives very good advice when he says to ‘buy the backup you need now, not in 18 months.’ I spent months looking into backup systems, including expensive raid arrays and JBoD boxes and home servers etc… That was because I was focussing on scalability more than cost. But all those solutions were too expensive, at least for me. I couldn’t justify the expense. So I thought about what I needed right now. And that was basically 1TB of storage. The quickest and cheapest solution would have been to just buy a cheap internal drive and an enclosure, and hey presto, a 1TB USB external drive. But the build quality of many cheap enclosures is suspect, and for a very similar price, you can get a entry level external drive. Then there is the question of interface. USB 2.0 is the basic requirement, but there’s also Firewire 400 and 800 and eSata (forget NAS devices for the moment).

In the end, I decided to take a small step up and get a WD My Book Home Edition. It has a triple interface (USB, Firewire 400 and eSata). I finally realised that there would be no instant implementation of a perfect system, and that it would be something I developed over time. My immediate requirement was to safely secure my data. I already have a 320GB external I built myself. That is now my secondary off-site backup, which is placed at a neighbours house. When I do a shoot, I backup to the 1TBdrive, and then go and collect the 320GB drive and backup to that, and then return it to the neighbours house. There’s my data in 3 places, one of them off-line. That is a secure backup.

But then there’s my system drive. At the moment I have my system drive ina  RAID array (Raid 1- mirroring). This is a great solution in the event of a total drive failure, as I have an exact copy ready to go. But what about OS problems? Recently I had a terrible security failure in Vista, which meant that the OS was basically kaput and needed to be re-installed, something I didn’t want to do. All those programs, all those settings… But I realised my mirrored drive now had exactly the same problems. Hmmm… So it seems to me, I need a complete mirror image of my drive to be backed up to an external drive whenever I make any major system changes or updates, and to have that stored offsite as well. So that means another external drive, and a program like Norton Ghost to make a mirror image. That has to come soon. Then, I should feel a little more confidant, that when (not if) a drive fails or the operating system decides to die, it will only be a new drive and a few hours, and all will be back as it was. That’s the idea anyway. Am I missing anything?

I’ve been deliberating lately about data backup, and how to design a solid strategy for both OS and data. All on a tight budget of course…

Being a photographer and avid net user, I can fill up hard disks with the best of ‘em. And having a ‘hoarder’ personality type also means [...]]]>

Backup man!

I’ve been deliberating lately about data backup, and how to design a solid strategy for both OS and data. All on a tight budget of course…

Being a photographer and avid net user, I can fill up hard disks with the best of ‘em. And having a ‘hoarder’ personality type also means that I don’t like to delete anything. So backup is a subject that has always been dear to me (see my earlier post on backup). But given that I’ve been a pc user for 15 years, I’m doing pretty well. Despite 4 hard disk deaths in that time, I’ve only ever really lost vital data once. And that just reinforced a lesson I already knew. So to me, backup means double or triple redundancy, with one set off site. But that’s in theory. In practice I have a more complicated and not quite complete set-up. Budgetary constraints shoulder most of the blame for that, but now I have decided that it’s time for no more excuses!

So where do we start? Because it’s not just static data we need to protect is it? Considering the nature of software (and here I’m talking both OS and applications) it is now a really big deal to do a clean install. An OS will need a huge number of updates (I’m a Windows user here) and then there’s installing all your apps and configuring the system the way you like it. So to me it seems pretty clear that a backup strategy has to take both OS and data into account. It’s figuring out the right and most cost effective way to do that I find tricky. And we’re never really starting with a clean slate are we. We work with what we have (in the way of drives and inteface options etc). It’d be a far sight easier if we were starting from scratch and had an unlimited budget. But hey, I live in reality-land down here.

My first stab at it led me to install a third SATA hard disk and set the OS up in a mirrored RAID array. My thinking behind this was that in the event of a hard disk failure, I would be up again in a very quick time, with no need to re-install the OS, or my apps. With a small business, I figure this was the basic setup for some sort of reliability and insurance against downtime. For my data I just used a USB external drive. I purchased SyncbackSE for my backup software and finally, burning all my raw files to DVD and storing them offsite completed the picture.

And then I had Vista break on me. Not a hard disk crash, but an OS disastrously corrupted. Irreparable it seemed. Suddenly it dawned on me that my RAID Array was not as useful as I’d thought. Now I just had two buggered OSes. And my single backup drive was getting terribly full. I ended up repairing the Vista installation after a great deal of effort, but it led me to re-examining my whole strategy.

The options are quite endless and equally bewildering. My first thoughts about the OS side of things had me thinking that ‘mirroring’ the drive would be a more sensible option that a RAID array. That would at least have saved me from my recent predicament. But in the event of a total hard disk crash, how up to date would my mirror have been? At least with RAID you don’t have to remember to write a new mirror every day (or every hour). So, pros and cons there, and I’m still not sure what’s the best approach. There’s probably good free mirroring software, but at the moment I haven’t gone that way as my drives are still in a RAID array.

And then there is data. A largish (1TB) USB2 External Drive can be had very cheaply these days, with a dollar to megabyte ratio of about 1:3 or 1:4 (ie. $250 gets 1TB) But there are obvious downsides to that approach. They are stand alone products that, like everything else, wants a USB port, and even with a recent computer, I’m running out of them. And there’s no sharing over the network or any of the other cool functions of a NAS device. I’m also trying to think into the future. Being a photographer and knowing that RAW files are 8 -12MB each (depending of course on the camera – even 21MB I think for a 1Ds MkIII), it’s clear that scalability needs to be addressed. And then we start getting into the budgetary constraints area. I’ve had a look at a few of the cheaper NAS options, and I’m starting to think that’s the way to spend the money. Something like the Netgear SC101 and a few 1TB drives… But it’s a jungle of price and features. My brain is shutting down with information overload and I’ve only been looking at this possibility for a few hours.

So. Any good ideas? I’d love to hear what other high data volume people do. Photographers, movie makers, download kings, whatever. I need some fresh perspective on the whole backup idea.

And of course, we haven’t even touched on online backup…

On Saturday my daughter was allowed her half hour on my computer to look at kids websites / games. She was on the Danish Radio web site [...]]]> With computers, the bottom line is that you can’t have enough backup. I have learnt this many times over the last 14 years. But I still keep learning it…

On Saturday my daughter was allowed her half hour on my computer to look at kids websites / games. She was on the Danish Radio web site when she called out to me that horrible phrase “Daddy, your screen is all blue”

Upon rebooting, there was a distinct clicking noise coming from the box in the vicinity of the hard disks. “Uh oh” I thought as I tried to remember when I’d last done a thorough backup. I had a long computerless weekend as I waited until Monday for the computer shop to open. I took it in, already knowing deep down that my disk had crashed and that I’d lost whatever I hadn’t backed up. I was guessing about 3 weeks.

I was right. The disk was dead, and while I’d backed up certain things like photos very recently, I hadn’t done a thorough backup for about 3 weeks. Bugger.

So $600 later I’m back up. Why $600? That’s a lot of money. Partly it’s because the shop I used charge like a wounded bull. There’s $250 of labour in that. They charged me $75 just to clone all the contents of my second drive over to a new one (long story but I wanted the data to go on a 500Gb drive and leave 2 x 250GB for the system drive in a Raid Array.) Then there was the cost of a new drive and the cost of Vista, since I decided to upgrade at the same time.

After a day of restoring backups, I have discovered the full extent of the damage. 3 weeks of emails – gone. This is OK considering I have emails on my pc that go back to 1998. 3 weeks is bearable. I haven’t as far as I can see, lost a single image. But I did lose loads of assignments I’d spent almost a week on, and I almost lost a website I was working on (luckily I’d been uploading it to the server as I went).

The moral is this. Don’t be complacent about backups. Even when you are serious about it like me, it’s easy to get caught. For me, it’s mostly the financial burden of setting up the proper architecture. For good backups you need a foolproof, automatic system. Even after $600 I don’t have that, but I’m a step in the right direction.