Like most (technology-oriented) parents, Christina and I have recorded hours and hours of Digital Video (DV) of Anya and Kyan growing and we’re still doing it now.
Digital Video (DV) requires a lot of space (of the order of 15Gb per hour!) Consequently, we edited the raw footage like mad and discarded all the crap to come up with beautiful movies (for us at least — I guess that most of you will fall asleep if you were ever subjected to them…)
Initially we only had one way to archive the movies: on VCD (No! Not DVD!) The reason is that we didn’t have DVD burners then. Of course, the quality is not great (MPEG-1 /Â 352×288 /Â 1150 kbit/s) but we personally do not care. Watching Anya discovering the world for the first time is what is important!
Then we acquired a DVD burner and shortly after we were producing our own DVDs. The quality level rose abruptly (MPEG-2 / 720×576 / around 2500 kbit/s) and we did that for years until H264 came.
H264 is the new MPEG-4 Part 10 video compression standard which has become pervasive since its adoption by Apple, Sony, Nokia and the Blu-ray Disc Association. Consequently, this year, Christina and I stopped creating DVDs and started rendering our movies to H264 at 720×576 and 1500 kbit/s.
The big decision
Two weeks ago, Christina and I decided that having VCDs, DVDs and H264 Quicktime movies on my MacBook was going to hurt us in the long run. I did a little bit of investigation and realized that we had to convert all our VCDs and DVDs to H264. We settled on using HandBrake to do the conversion because, well, it’s excellent and open source.
As we are planning to buy an AppleTVÂ some day, I used the AppleTV preset in HandBrake with the following changes:
- H264 Video / 720×576 / 1500 kbit/s
- AAC Audio / Stereo / 48kHz / 128 kbit/s
- 2-pass encoding
- Turbo first pass
- Anamorphic: no
- Keep aspect ratio: yes
- Crop: manual
- Deinterlace: yes, slow.
The result
Christina and I are now the proud owners of 48 home made H264 movies taking 17Gb of space (more than 24 hours.)
We have bought an external hard disk (a Lacie) to store those 48 files and I’ve copied them on the hard disk of one of the desktop computers I have access to. I have also copied them on a second external hard disk I normally use to backup my MacBook. All in all, I’m happy… for the time being.
Amazon S3
I have been looking at Amazon S3 for some time now as a means to keep my most important files in a data-center somewhere. And those 48 movies certainly qualify as my most important files ever! Unfortunately, sending 17Gb of data to Amazon from Mauritius is not practical (and this is one of the most massive understatements I’ve ever made) due to the pathetic bandwidth we have here.
Let’s see, at 128 kbit/s, 13 days would be required to upload all the 17Gb of video. Maybe it’s more sensible paying an air ticket to Réunion Island and doing it from there… Anyway, the cost to keep the data on Amazon S3 would then be:
- $4.25 (Rs. 113) for the first month (to account for the initial data transfer)
- $2.55 (Rs. 68) per month thereafter.
This is extremely cheap! One added benefit is that the videos will all be accessible online. Surely, Kyan and Anya will be able to stream the movies from S3 to their mobile phones when they’ll be big enough to have phones :-)
A question
Have you had the same problems? What strategy have you settled on for archiving your precious digital memories?
The only files that I preserve are my pictures. I make several copies of them on DVDs in a staggered manner. For example, collections 1 to 35 on DVD #1, collections 15 to 50 on DVD #2, and so on. It’s a poor-man’s RAID, but that will have to do until we get ZFS in Mac OS X, at which point, I will back up to an external HDD — I have seen too many good HDD fail to trust them fully.
Eddy.
I backup my iPhoto collection on an external hard disk and I keep the best online (albeit at a much lower resolution) using Flickr.
Unfortunately, video files still require too much bandwidth to be stored online for the time being. Of course, I could resize them to, say, 352×288 (VCD size) but then I believe the quality will be degraded too much… I guess I’ll have to wait for some additional years. Meantime, I’ll rely on multiple hard disks… Thanks for the tip on ZFS. It seems that the next version of Mac OS X will be ZFS-based.
ZFS looks very promising. If you haven’t read the details, this testimonial from James Gosling should be convincing enough.
Mac OS X Snow Leopard Server will bundle ZFS apparently, but nothing has been said if the desktop Mac OS X will.
Eddy.
A good backup plan without an equally good restoration plan is worthless.
Eddy.
All my photos & videos are on dvd… awaiting to be burned on Blu-ray (when commercially available & viable). :P
As for online backup storage, I would recommend Xdrive – 5GB & free! ;)
As for x264, I believe that it won’t be long before it will replace xvid as the main video format.
I’ve been toying around with MeGui, & the results were great. Half-size of xvid & greater quality. :)
I’ll check out Handbrake…
What do you mean Eddy?
Files copied to an external hard disk are as easy to restore (just access them) as files found on your internal hard disk. As for Amazon S3, it’s possible to access a file using a URL. I’ll say that restoration is pretty much taken care of.
@Avinash: I was referring more to 4lyf’s post than yours, what with the reliable-but-complex ancestry backup method.
External HDD are convenient but prone to failure, so HDD backups require frequent verification to ensure that the files do not go corrupt without you noticing.
Eddy.
@Avinash
Backing up on a remote site is good because in the event of a catastophe (knock on wood), you won’t lose all your invaluable data. However, there is no guarantee that a company won’t go out of business. Zero risk does not exist.
I am currently evaluating different strategies to backup my files. One way for me to have a remote backup would be to burn my most important data to DVDs and them store a copy at work.
@Eddy
You’re right. A good backup means nothing if files cannot be restored properly. Time machines and iBackup failed some of my test in the past. That’s why I don’t trust them and am currently leaning towards Carbon Copy Cloner. You’re also right to say that HDD are fragile. I, too, have seen so many of them went south.
Hi Patrick,
I have been using SuperDuper! for two years now and I’ve been very satisfied.
It has even saved my ass once when I lost the internal hard disk of my MacBook.
It is free for basic use. But you only get incremental backups when you register.
Am I the only one who hands it all to fate? *grin*
Seriously, I cannot wait for ZFS to be part of Mac OS X. One cool thing I would be able to do is making a storage pool with my internal and external HDDs as mirrors. Then, whenever I connect the external drive, ZFS will sync it to the internal one automatically.
Eddy.
@Avinash: It looks like Askimet flagged one of my comments as spam wrongly.
Eddy.
Hello Avinash,
I tried SuperDuper once, but did not like it very much. It’s funny, because SuperDuper and Carbon Copy Cloner are almost the same, except the first one is not free, whereas the second is. I also like the fact that Carbon Copy Cloner’s writer wants you to only donate when you completely trust the product and it does not block some of the features in the free version (there’s only one version).
Does SuperDuper do a real incremental backup or a differential backup? CCC calls its backup “incremental backup,” but it seems to be doing a differential backup where it will backup only files that have changed, unless of course, I have not clearly understood the difference between differential and incremental backups.
@Eddy
Some like to live dangerously :-) I’ve been pretty serious about backing my data for a long time, but the most important reason is probably of what happened to me in Dec. 2004. Two days after having submitted my last assignments, my HDD died.
@Avinash
I think Askimet flagged my last comment too.
@4lyf: I hand it all to fate, remember?
Seriously, if the Apple computer burned, I’d 1) buy a new one, or 2) recover the contents of the HDD from anothor ZFS-enabled operating system.
@Patrick: Most of my works, except my CV and PGP keys, are in the cloud. My code is in CVS. My MP3s can be ripped again or downloaded. Same for movies. There is very little for me to back-up actually.
Eddy.
hell you are kinda picky on quality!
I usually convert everything to VCDs and save em on DVDs, save space and i usually save two copies of everything and back whatever is classified as highly important in several 3 DVDs, 1 on my normal computer, 1 on my file server and 1 copy where i host my website [300GB i’ve got to use that one day or the other] :p.
if it is even more important than that, then i encrypt it and save it in the above locations.
since i upload immediately after i find sometihng important [reason being: i am parano and unlucky when it comes to things blowing up], i never really feel the load big.. :p though after that, i never really would want to download what i uploaded, then i would prefer to waste the entire day searching dvds that looking for it on my host.
definitely, you are sick to be that crazy about quality for backup’ed data.
I was referring to “archiving” in my post. Not “backup” :-)
@Avinash: Somehow the backup-related issues that are being discussed also apply to archiving.
I know archiving focuses more on accessibility whereas backup is mostly about safeguarding, but the reliability of the archival media is relevant in both cases.
Eddy.
I usually achieve all my documents, photos, videos and audio on DVD-Rs. As a safety measure, I make two copies on two different brands. It’s not the most practical and fastest solution, but it works for files less than 4.2GB (or less than 8GB for DVD-DL).
I tend not to trust hard drives for long term storage. In the past, I’ve had several drives failing after less than a year of moderate usage, leading to gigabytes of unrecoverable data…The prices for mass storage have considerably gone down, and unfortunately so has the quality/reliability in my opinion.
Just like video, uncompressed audio data and multitrack audio takes an awful lot of storage space; especially when working in stereo/24-bits/48kHz or 96kHz…In such cases, I usually have a second external hard drive solely as backup. After each session, it’s much faster and easier to copy all the contents from my dedicated audio drive to the second drive. I don’t have to worry about synchronizing the data contained on each volume or spliting its contents on multiple discs. At the end of a project, I usually burn them to multiple DVD-Rs (in both native and AIFF format).
I think the most problematic issue when archiving data is that of changing formats, standards and media in consumer electronics. Will the videos encoded today in whatever current MPEG standard be available in 20-30 years? Will the documents I’ve created in my favorite word processor (or any other proprietary format) in 1990, be readable in 2010? Will files compressed or split to fit multiple discs be still be retrievable in the next ten years? Or will the optical drives we use today be slowly be replaced by solid state drives (SSDs)…just like the audio CDs replaced vinyls and compact cassette tapes.
Over the last ten to fifteen years or so, de-facto standards and open formats like PDF, RTF, JPEG, WAV, AIFF, MPEG etc…have worked pretty well. Let’s hope the programs we’ll use in the next decades will still support those formats on which we’ve come to rely…