STORING YOUR DIGITAL IMAGES
Over the last six months the topic of archiving digital images has been high on the priority list of those who have attended my workshops and trips. With the rapid maturing of digital camera technology and the ever increasing files sizes that each camera generation can deliver, the need for a good storage and image retrieval system is becoming every more critical.
Unlike archival film storage systems which were designed to prevent or minimize degradation due to age, digital images must fit a different set of criteria. Unlike film, digital images do not generally show this gradual decline. Any loss tends to be immediate and catastrophic. You can either read a file or not. The disk you used last week without problems can suddenly refuse to work. You need to plan your digital storage to take account of this difference.
While no one storage system or technology is right for every photographer. There are some fundamental issues that must be considered to help determine which system or technology to adopt. These are:
Capacity;
– Number of images (both current and projected) that are to be stored;
– Size and type of images to be stored (e.g. low resolution jpeg’s, RAW + jpg, RAW + converted file [.psd, .tiff, etc.])
Manageability;
– The easy of locating a file and accessing it.
File Handling/Flexibility;
– Will a given file need to be modified in the future or will it never change.
File Preservation and Long Term Safety; and
Budget.
In selecting a storage system, you need to take into consideration both your own needs as well as the different characteristics of digital media storage systems that are available today. You have to think through the design from when the original files (digital negatives if you will) are stored on a card in your camera to perhaps twenty years on when you will want to locate an image in your storage archive.
Even with computers now containing 80-120GB hard drives, it is easy to run out of storage space thanks to today’s larger files sizes and more images being shot. One answer to this problem is to store the images off of the computer’s primary hard drive and retrieve them as needed. One of the most cost effective means to store images is to put them on optical disks such as CD’s or DVD’s.
CD/DVD Backups
These provide a useful backup and work reasonably well if the number of images you have is small. Once the disk count get to be over 30 units the task of managing and locating images becomes much more difficult. Another problem is that disks are not 100% reliable. If you use them, always buy good quality branded disks, and have suitable storage for them. Probably the best storage systems are in cases protecting them from any bending or creep.
It is probably best to write a whole disk in one session and to fix it so that it can no longer be written to. Use CD/DVD writing software that will verify every bit on the disk after it is written, and discard any with errors. Even with these precautions, the materials which are used to make CDs and DVDs decay over time and are highly susceptible to surface scratches and cracks (from bending) that can make the whole disk unreadable. Different brands and different grades of optical media advertise different life spans. It also depends on how they are stored, how well you take care of them, how often they are read, etc.
With even a simple scratch rendering your CD or DVD unreadable, it is advisable to make multiple backup copies. In the past, when I had a smaller library of images, all of my master files would be stored on three sets of CDs (later DVDs). One set (Set C) would be stored offsite for safety while the other two sets remained in the office. Set A would then be used exclusively as the working set while Set B was not touched and served as the backup to Set A in case of data failure.
This system works well up to a point, especially if you have a small library of images to manage. While an optical disk system is both economical and provides a fair level of security for your images, it lacks flexibility if you have to modify an image later on. It can also become difficult to organize, manage and retrieve specific files as the number of disks grows. As I continually learn more about the post-camera conversion of images, I have the tendency to go back and refine an image months after I had created a master file. (That is why I always save layers as part of the file.) Every refinement created another “version” of the original image and necessitated having to go back and burn three new disks so that each would have the “current” version. You can imagine the hassles this process creates if you keep multiple images on a disc.
HARD DRIVES
While storage space is one of the primary reasons behind forcing the expansion of any image storage system, the short term answer is to simply add more storage capacity such as an additional hard drive. While adding a second hard drive to a desktop system is possible, doing so on a notebook computer is not. As such, I will focus on external drives for the remainder of this article.
Relative to optical disk storage systems the advantages of hard drives are numerous. As technology has advanced the cost per MB of storage space has fallen significantly. Image management is much easier especially if you are using on of the many image management software packages that are currently on the market. Also, file retrieval is faster as the data transfer rate is greater for either USB 2.0 or Firewire connections than a CD drive read rate (not including the time it takes to go and find the correct CD/DVD in the first place).
On the down side, digital files are notoriously easy to lose. All of us have deleted files in error, and anyone who has owned computers for some years is likely to have had hard disks and other storage media fail. As such the issue of file safety must be taken into consideration.
Single Drive Systems
In today’s market, external hard drive systems fall into two categories – single hard drive systems and multiple or RAID systems. Of the two, single, stand alone, external hard systems have been around the longest and command the larger part of the external storage market. There are too many to mention here but as of this writing, one can purchase plug and play external hard drives with capacities up to 750 GB. (Think 75,000 10 MB jpeg images or 6,000 125 MB tiff images. That a lot of shooting!)
Single disk hard drive systems have all of the benefits listed previously for an image storage system. The problem is, like the hard drive in your computer, it too is subject to head crashes and subsequent disk failures. (Note: unlike failed optical disks, data can be recovered from failed hard drives but it is an expensive and time consuming process.)
Multiple Drive or RAID Systems
Though multiple hard drive technology has been around for awhile, recent developments have brought the costs way down as to be affordable by the consumer market while making it easier for the average person to incorporate it into their home computer system.
As external hard drive capacities swell, manufacturers recognized that the more that can store, the more there is to lose. This is where RAID technology comes in. RAID stands for Redundant Array of Independent Disks. A RAID system is simply two or more hard drives yoked together and controlled either by software or a circuit board in the external hard drive housing. Of the two control methods, I prefer the latter as there is no installation or set up time required to get your system up and running. Simply plug it into an external port, assign the system a drive designation, add folders and you are ready to start.
RAID systems use multiple hard drives to achieve one of three goals: improved performance, improved redundancy (duplicate data), or both. A system with multiple drives can yield better performance since data can be read from more than one drive at once. A RAID system can also achieve data redundancy by using some of its storage capacity to mirror data that can be used to rebuild a RAID array in the event of a drive failure.
For the purposes of this article, RAID systems fall into two general configurations – RAID 0 and RAID 1 (there are more types but they are hybrids of these two).
RAID 0
A RAID 0 configuration turns multiple hard drives into one logical drive that your computer sees. Data written to this array of drives is spread over multiple disks, which improves performance and can greatly increase storage capacity of your system. Most commonly external RAID housings can accommodate either two or four hard drives. This means that a four hard drive housing containing 500 GB drives will be seen by your computer as being a single 2 TB (that’s TERA bytes) drive.
However, while having the potential to greatly increase storage capacity, the issue of data reliability and protection is the same as it is for your computer’s hard drive. A multiple drive system in a RAID 0 configuration is vulnerable to the failure of any one of its physical hard drives. If, for example, you had a two-drive RAID 0 array, your mean time between failure rating (MTBF) is cut in half since you have twice the exposure to drive failure. A failure of any drive in a RAID 0 array will kill the array completely and take with it all the data on the logical drive.
RAID 1
A RAID 1 configuration (also known as mirroring) solves the data safety issue present in a RAID 0 configuration. In RAID 1, two hard drives work together as a pair where each has identical content. Thus, each hard drive of the pair mirrors that other. Despite the fact RAID 1 uses two physical hard drives, the operating system on your computer only sees one logical drive. Since data on one drive is duplicated on the other, a two-disk RAID 1 array’s storage capacity will only be equal to the storage capacity of a single disk. The advantage of this configuration is that in the event of a drive crash, just replace the damaged disk with a new drive, and the RAID card will rebuild the mirrored array. This is a far less expensive way of restoring your data than sending the crashed drive out to have the data recovered. Raid 1 is relatively cheap to implement and gives useful protection. It doesn’t affect file-writing times, but speeds up reading from disk which is useful when you work with large files in Photoshop etc.
There are a growing number of RAID external systems being offered in the marketplace by manufacturers such as SansDigital, Buffalo Technologies and Lacie. Personally, I have installed a RAID 1 system made by SansDigital as my next generation storage system. Products offered can comes either with the hard drives already installed or you can purchase a separate RAID housing (they come with cables) and then purchase your own hard drives.
If you go this latter route, be advised to install hard drives that are the same capacity, and model and which are made by the same manufacturer. This avoids potential problems caused by having two drives with dissimilar read/write speeds or performance capabilities. Hard drive installation into the housing is easy and straight forward as most systems are designed to have the hard drives slide into a slot and snap into place. My total install time for the SansDigital was about 20 minutes and included installing the two disks into the housing, connecting to the computer and giving it a drive configuration so that the operating system could recognize it.
Separate Storage and Backup Strategy
Regardless of the system you setup, there is little use in having two (or more) copies of your images if you keep them both in the same place. In case of fire, flood, line surges or other catastrophe you could simply lose both copies. (Note: in the case of external hard drive systems you should install a good [not a $4.95 Wal-mart special] power surge protector between the wall outlet and the computer and external drive. Line surges caused by lightning strikes [even miles away] and blown transformers can fry circuit boards in a fraction of a second.)
If you have separate work and home locations, then it makes sense to keep a copy of your digital negatives at each place. If not, think of a friend or relative who you could trust to keep your copy safe. Perhaps you could find another photographer and arrange to keep each other’s images safe. There are also a growing number of internet-based, image storage sites where you can park your images.
Apart from the internet storage option, you have three technologies available that will allow you to create a backup to your data depending upon the storage system you choose. The first is optical disks in the form of CDs or DVD. The second is tape backup systems. These have been around the longest, are relatively inexpensive and reliable for short storage periods (tape will deteriorate over long periods of time just like film). If you have a RIAD 1 system, the third alternative is to purchase a third hard drive which can be used as the off site storage vehicle. To back my data up, I simply remove one of the drives and install the backup drive. The system then overwrites the drive with the new data. When complete, the drive is removed and stored elsewhere.
Whatever the technology or system you choose, it is important to maintain a regular backup schedule so that backed up data remain current and can be checked on in case of media deterioration.