Outgrown image storage strategy, ready to bite the bullet and do it right

[Steve Uffman]

Currently, I am using 3tb internal and 11 tb external (4 WDC mybooks). Part of the plan includes rotating two of the 3tb externals offsite......Data is mostly images but not only.

and tired of win7 intermittently not recognizing one of the externals unless I power of the external and power back on (occasional problem but irritant no less)

so looking for a best of breed suggestion that is expandable.

Have heard Drabo, linux solutions and more.

Good time to do it as one of my internal 1tb drives is crashing and to be replaced later in the week. So want to move on this.

Thoughts?

Thanks
Steve

[Ed Cordes]

I hope there are some posts to this question as many of us are wondering the same thing. Is an array of separate HDs the best. DROBO is an option but seems to be redundant data storage rather than true back up. Perhaps an enclosure that houses multiple HDs but manages separate access to each? I don't know either. So, I join Steve in asking for input.

[David Stephens]

What problems are you trying to solve. Looks like you have plenty of drives, but do need to get one off-site.

Why do you have such a large internal HD? That makes me think that you're not backing up often enough. You need to back up regularly, including off-site, and clear the internal regularly.

How large is your archive?

If you're staying with Windows and buying new drives, then look for USB3 to speed backups and syncs. Hopefully you've at least got Firewire 800 for the current setup.

[John Chardine]

My 3TB images drive (Seagate Goflex Desk) contains about 83,000 images @ a mean of 17mb each and I am not half-full yet. I have neglected culling for a while and need to get on with it (83,000 images, most of which are RAWs is too many!). I have two other 3TB Goflex drives on which I alternate Time Machine (Mac) backups, one of which is always offsite and the other is in a safe at house when not in use. I have a modern Mac with Thunderbolt (TB) and just converted the drives to this standard by purchasing Seagate TB bases for each. TB is much faster than Firewire800 and USB3 and glitch-free compared to the old USB2 which was slow and unreliable (drives would occasionally disconnect on their own). This system has considerable room left and is scaleable as I add more TB drives, which are daisy-chainable up to a limit. I looked at RAID but decided that my backup system obviated the need for the redundancy of RAID 1-6. I consider a important but neglected part of my image management system to be aggressive culling. BTW TB is an Intel (with Apple) technology which is starting to show up on regular PCs.

An important part of a scaleable system is image cataloguing software which creates and manages databases of images on n volumes, none of which need to be online. I use Photo Mechanic which does not catalogue yet, but is promised to do so in the next while.

[Greg Basco]

Steve, I'm about to update my system too but it will only be a variation on what I currently use. I do it pretty on the cheap as I didn't want to pay for Drobo and also didn't want to have something running and backing up all the time.

I currently have one 250 GB internal drive for the OS and then one internal 1 TB drive for images. I back these up once every few days (or as necessary if working on a project) to two external 1 TB drives (same data on each so I have the data in three places at this point). I unplug these and keep them up in my closet when not using them for security's sake (lighting, virus, theft). I do the backups using the free SyncBack software for PC. I also back up my more important images every week or so to a 500 GB passport drive, which I keep in a safe deposit box at my bank in the town where I live. So, pretty old school I suppose but quite cost effective and easy to do.

I'm getting a new PC next week with a 2 TB internal drive, a double hard drive enclosure with two 2 TB drives, and a 1 TB passport drive. So, more space but the same system. One of my sons, a budding photographer and graphic designer, will get the old drives.

Hope these ideas help.

Cheers,
Greg

[Steve Uffman]

1tb internal is os and programs. 2tb internal for data (mostly images and video)3tb externals rotated for backup each Sunday with one going offsite so pretty much 3 copies each. 2tb external has all videos
backup

17mb average image not true for me with 5diii and 1dx raw


Plus work files from ps

All USB 3.0 connections except for 2tb for videos

Likely will move images off internal to external if windows 7 issue with dropping external drives can be mitigated

Lr4 cs6 nik

[Steve Uffman]

Steve, I'm about to update my system too but it will only be a variation on what I currently use. I do it pretty on the cheap as I didn't want to pay for Drobo and also didn't want to have something running and backing up all the time.

I currently have one 250 GB internal drive for the OS and then one internal 1 TB drive for images. I back these up once every few days (or as necessary if working on a project) to two external 1 TB drives (same data on each so I have the data in three places at this point). I unplug these and keep them up in my closet when not using them for security's sake (lighting, virus, theft). I do the backups using the free SyncBack software for PC. I also back up my more important images every week or so to a 500 GB passport drive, which I keep in a safe deposit box at my bank in the town where I live. So, pretty old school I suppose but quite cost effective and easy to do.

I'm getting a new PC next week with a 2 TB internal drive, a double hard drive enclosure with two 2 TB drives, and a 1 TB passport drive. So, more space but the same system. One of my sons, a budding photographer and graphic designer, will get the old drives.

Hope these ideas help.

Cheers,
Greg

Simple but effective. Mine is not far from yours except I use 3 tbs to backup the 3 tb internal. I always resample the state of the art whenever I upgrade. Not sure I have seen a better solution

[David Stephens]

1tb internal is os and programs. 2tb internal for data (mostly images and video)3tb externals rotated for backup each Sunday with one going offsite so pretty much 3 copies each. 2tb external has all videos
backup

17mb average image not true for me with 5diii and 1dx raw


Plus work files from ps

All USB 3.0 connections except for 2tb for videos

Likely will move images off internal to external if windows 7 issue with dropping external drives can be mitigated

Lr4 cs6 nik

Hard to believe a need for 1TB for OS and programs. You must have some hellacious programs not affilitated with you imaging needs. Still, if you need it, it's cheap.

Still, I don't understand what's wrong. Sounds like you need to spend a week or two culling. Then develop the discipline to cull daily after each shoot. That's certainly the best time to cull. Keeping a bunch of 30MB Raw files (my 5D MkIII ranges from low twenties to mid-thirties, depending on lighting.

You've got plenty of capacity, so I don't see what you want to change.

I've been using Windows 7 64-bit for a couple of years now and don't experience dropped external drives. I use LaCie. Is this common to a particular drive. When I go to My Computer, my drives always show up. I drag and drop between externals with no issues.

[Steve Uffman]

Dave, thanks for response. 1tb internal purely os, programs ,documents such as e books and other and downloads. It has plenty of room but I always have kept os and most data files separate.

Lr has over 80k images on the 2 tb drive plus numerous ps work files. Represents great deal of family archives that were digitized after many family photos for different family members were lost 7 yrs ago in hurricane Katrina. Workflow flags culls as step 1. Significant storage goes to keeping ps tiff work files. Curious what most folks do with these. I know several pros that hang on to them on key images.

Glad you are not having win 7 dropping drives. It is a known issue in win 7. But not a problem in Linux which is an option.

By the way Dave, this is not my first rodeo. I founded a tech company when I was 28. When I sold it 7 yrs ago, our data center had over 800 servers providing mission critical service to nearly 500 financial institutions in North America . We were one of the earliest and most successful ASPs. Also housed a super computer. We had the highest developer status with Microsoft and the highest security operational status available to serve financial institutions .. And while most of Louisiana crashed during Hurricane Katrina, we never had a second of downtime due to our level of redundancy and planning. So just maybe I am not totally clueless

Now back to this issue, my mo is that storage and, memory are cheap. Whenever, I consider an upgrade I always survey this site and another to see if there is an evolving best practice. Also the number of days in the field is accelerating with my wife and I both taking images. Galapagos, 2 trips to Alaska, 20 days in Africa, Minnesota black bears and more in last 18 months. More to come with polar bears and likely Antarctica in 2013-So growth in capacity is certainly a design parameter

Anyway, appreciate your interest

[Greg Basco]

Steve, good to hear I'm doing my backups in a similar fashion to someone with your experience! I've not had problems with Win 7 dropping drives but, as I said, I generally don't leave them connected but rather connect and disconnect as necessary when I'm doing/finished with a round of backups.

Cheers,
Greg

[Steve Uffman]

Steve, good to hear I'm doing my backups in a similar fashion to someone with your experience! I've not had problems with Win 7 dropping drives but, as I said, I generally don't leave them connected but rather connect and disconnect as necessary when I'm doing/finished with a round of backups.

Cheers,
Greg

Greg simple is often the most elegant if it works. My bet is the win 7 dropping drives periodically is a function of the long periods of inactivity since they stay connected but pretty much are only active during backup or reboots your process would make that moot.

My problem manifests itself most often when the unattended backup process starts and cannot find the designated drive anymore. A reboot and restart then all is well. We have investigated thinking it was related to USB ports going to sleep and being dropped but setting changes did not make much difference. It is an infrequent problem but an aggravating problem.

Curious what you do with the huge intermediate tiff files out of PS? I know some keep them but that practice gobbles huge amounts of storage per image if you keep many of those.

In other matters, the 5diii was terrific in low light situations in Africa. I rented a5diii for my wife, coupled with a 70-300 that you recommended (along with others). I would usually set her up in auto ISO ( within limits) and then a reasonably fast ss and wide aperture and she did phenomenally well. The ISO ranges were never wildly high like I saw on the 7d. my 1dx also was superior in low light as predicted.

[Greg Basco]

Hi, Steve. Good deal. Thanks again for the confirmation. I actually don't produce many TIFF or PSD files. For my wildlife work, my RAWs are my masters as I do all processing in Lightroom. I have started to produce some TIFFs and PSDs of landscape files, however, for the coffee table book on which I'm working. Landscapes definitely need some layered processing attention. That's the main reason I'm upgrading my PC and backup system actually is the big files for this project and the fact that I'm starting to get more and more into landscapes.

Glad you had a good trip to Africa!

Cheers,
Greg

[David Stephens]

No problem Steve, which seems to be your problem. No kidding. I don't see where you need an upgrade. If everything is working, then what are you looking for?

It seems to me that the next step is to consider co-location. You've got the connection that you should be able to get some virtual server space for a "relatively" nominal cost.

[Steve Uffman]

No problem Steve, which seems to be your problem. No kidding. I don't see where you need an upgrade. If everything is working, then what are you looking for?

It seems to me that the next step is to consider co-location. You've got the connection that you should be able to get some virtual server space for a "relatively" nominal cost.

You are a "risk" consultant, right? For mortgage banking? You offer strong opinions with not full picture No kidding!

[David Stephens]

You are a "risk" consultant, right? For mortgage banking? You offer strong opinions with not full picture No kidding!

Oh, I see, you just want to blow some money. Call Accenture then.

[Steve Uffman]

My 3TB images drive (Seagate Goflex Desk) contains about 83,000 images @ a mean of 17mb each and I am not half-full yet. I have neglected culling for a while and need to get on with it (83,000 images, most of which are RAWs is too many!). I have two other 3TB Goflex drives on which I alternate Time Machine (Mac) backups, one of which is always offsite and the other is in a safe at house when not in use. I have a modern Mac with Thunderbolt (TB) and just converted the drives to this standard by purchasing Seagate TB bases for each. TB is much faster than Firewire800 and USB3 and glitch-free compared to the old USB2 which was slow and unreliable (drives would occasionally disconnect on their own). This system has considerable room left and is scaleable as I add more TB drives, which are daisy-chainable up to a limit. I looked at RAID but decided that my backup system obviated the need for the redundancy of RAID 1-6. I consider a important but neglected part of my image management system to be aggressive culling. BTW TB is an Intel (with Apple) technology which is starting to show up on regular PCs.

An important part of a scaleable system is image cataloguing software which creates and manages databases of images on n volumes, none of which need to be online. I use Photo Mechanic which does not catalogue yet, but is promised to do so in the next while.

Thanks john ,was familiar with the thunderbolt tech but had not seen it on pc . Will investigate . My backup scheme is like yours in how you alternate . Your 3tb drive for images is internal or external? And the disconnect problem you referenced with the 2 tb connected via USB is the same thing I have seen but on Usb3.0 as well. Apparently related to inactivity of the USB device according to some of the win forums. like you, I did not see any benefit to RAID over this scheme. Do want to investigate the scalability you reference

Am interested in how you treat intermediate psd or tiff files that can be huge if saved. I know some save. Others do not. I was not saving any of the interim files until I learned some pros do if they have multiple purposes or deliveries. I always would flatten and then save. Suggestions on your workflow on such images would be welcome


also Have been using only Wd drives but ran into the formatting issue on the 3tb drives for Windows. Soured me a bit on the WD quality control. Also the WD software had to be disabled as it was constantly grabbing large chunks of memory. Do love the passport drives though

[Roger Clark]

I'll give some of my methods. First, sorry for not responding sooner--I just got back from Alaska where I had no time for the internet.

First, I never re-use an old disk drive. 2 reasons: 1) When I take an old drive out of service, I consider it an archive backup and put it on the shelf. 2) Re-using an old drive is in my opinion dangerous as an older drive may be more likely to fail.

If one used a rotating cycle of disk drives, one could delete or mess up some files/directories and if not discovered, the rotating backups would propagate that mistake. Thus by making old drives an archive backup, that forms insurance against at least some problems. So whether an internal or external drive taken out of regular service, it goes on the shelf as part of my archive. (If one can keep at least some of those drives off site, even better.)

For rotating backup, check out synctoy from microsoft (free):
http://en.wikipedia.org/wiki/SyncToy
http://www.microsoft.com/en-us/downl....aspx?id=15155
(check if this is the latest download).
It is highly rated (but I have not used it).

For years I used rotating USB disk drives to back up my windows xp photo desktop. I would rewrite an entire drive (I wish I knew about synctoy).

Then, after one computer upgrade, I bought a new internal drive for my old computer and installed linux on it. With Samba, I could mount a disk on the linux machine on my windows desktop and copy files to the linux box from windows. Worked great, so I could back up files at will without having to keep a USB disk plugged in. I still did USB disk rotating backup with some disks off site.

In 2011 I went to Africa with a windows 7 laptop. I had win7 for only a few weeks. I hated the win7 interface. It wanted to do things, but not what I wanted. I swore I would never return to Africa with windows. After that trip, I returned and put together a new desktop for photo work: Ubuntu linux 10.04 with rwo internal 2 terabyte drives, a 1.5 TB system disk, quad core I7 with 12 GBytes ram. For photoshop I installed virtualbox and windows 7 where I installed only photoshop and ImagesPlus (another image processing system). A clean windows runs quite fast. A second system acts as a backup server. I had never used the graphical interface of linux for everyday work until 2011 (but had used linux/unix command line for decades). What a surprise. Ubuntu linux was much like the XP interface with intuitive improvements and more functionality! I now use linux for most of my work, including presentations, teaching classes, writing papers, email and web. I only use windows when I run photoshop or imagesplus.

August of this year I went to Africa with a linux laptop. No windows and no photoshop (didn't have time to install photoshop in virtualbox). Very impressive. The standard file browser (nautilus) starts offloading files from the card and I could immediately start viewing images as the disk filled with no rehresh required. Nautilus even does raw file viewing. The standard image viewer is fast to view images with better functionality than other tools I've used on windows for zooming, panning and moving to the next image, as well a better functionality on small screen laptops (windows 7 programs seem to take up more screen real estate with their frames).

I do my backups with rsync. I create a one line command file to do the rsync and execute it whenever I want (or automatically, periodically).

I'm a firm believer in solutions that are not proprietary as I want to be able to have the system working long into the future. Example, I have a spectroscopy database that gas been operational with essentially no maintenance since 1984 and the database gets updated automatically every day with hundreds to thousands of new entries. It is a non-proprietary solution and has worked on unix and linux, with the only maintenance of copying the programs and database to a new computer (running linux or unix).

For those who want to try linux, Ubuntu has moved to what is called the Unity interface and is being geared for tablets. If you want a more xp-like interface, try Linux Mint with the Mate interface (that is what I am moving to). With Macs running unix, Android is linux, linux on desktops, most supercomputers are linux, most servers are linux, the phone system is unix, tivo is linux, etc, linux/unix is mainstream and on more devices than windows. There is a rumor that Adobe is working on photoshop for linux.

Roger

[Ken Watkins]

Steve,

Having used external hard drives for years I found them very frustrating, so I bought an external server to store everything I got a QNAP with 4 x 2TB Hitachi 2TB disks, and touch wood it has worked so far, although Windows sometimes cannot recognise it is there

The only problem seems to be the speed of transfer, but that is probably down to my "African" router

[Steve Uffman]

Thanks Roger and Ken for the responses . Lot to consider but kind of info I am looking for. On the speed of transfer, Thunderbolt as John C. alludes also has some promise to improve transfer rates but after some late night research on adoption in the pc world looks to be non trivial if the manurfacturers do adopt it . Dell has not jumped on board.


Thewin7 environment has been more support intensive than I would like both on my desktop and notebook. But that maybe the result of a dreaded root kit that hit my desktop. Tough calculating the residual impact of that even though I have rebuilt both systems with disk reformats and operating system reinstalls. I will look hard at your solutions. Thanks again for your always useful input

[Steve Uffman]

On the subject of file sizes and what you save out of PS-Do you do your adjustments in PS, Nik and etc. then flatten in PS before saving as a PSD, Tiff or whatever? or do you often save the intermediate file which can be HUGE? Is so, any particular way you save it?

Trying to figure the optimum workflow and if saving the intermediate file on those special images mean a Huge Tiff file with layers intact, then I want to be sure to plan for it. Since most often my images go to the web, I usually flatten, automate to file size, save for web and the publish. More recently, I have had folks ask for prints so that will change things if the image is delivered in separate mediums, right?

Thanks

[John Chardine]

Thanks john ,was familiar with the thunderbolt tech but had not seen it on pc . Will investigate . My backup scheme is like yours in how you alternate . Your 3tb drive for images is internal or external? And the disconnect problem you referenced with the 2 tb connected via USB is the same thing I have seen but on Usb3.0 as well. Apparently related to inactivity of the USB device according to some of the win forums. like you, I did not see any benefit to RAID over this scheme. Do want to investigate the scalability you reference

Am interested in how you treat intermediate psd or tiff files that can be huge if saved. I know some save. Others do not. I was not saving any of the interim files until I learned some pros do if they have multiple purposes or deliveries. I always would flatten and then save. Suggestions on your workflow on such images would be welcome


also Have been using only Wd drives but ran into the formatting issue on the 3tb drives for Windows. Soured me a bit on the WD quality control. Also the WD software had to be disabled as it was constantly grabbing large chunks of memory. Do love the passport drives though

Steve- The 3TB drive is external. The only internal drive I have is a 250gb SSD drive in the Mac. It is fully loaded with all the software I need and my music collection and is way less than half full. These days you don't need a big HD for system and apps.

Re. the intermediate files, I save them with layers intact so that I can go back to them and save the processing time. Because I only process selected images, I am not concerned about their size. This is not a huge deal though because I don't do much to my images so going back to the RAW for re-processing is not a huge deal.

Agree the Passport drives are a great product. I've has a little 500gb USB2 Passport for ages and it has never hiccuped once. I really like the Seagate Goflex drives because of the interchangeable bases giving you a variety of connection options. A direct SATA port is not available on Macs but that's a fast option for connecting drives.

[John Chardine]

Thanks Roger and Ken for the responses . Lot to consider but kind of info I am looking for. On the speed of transfer, Thunderbolt as John C. alludes also has some promise to improve transfer rates but after some late night research on adoption in the pc world looks to be non trivial if the manurfacturers do adopt it . Dell has not jumped on board.


Thewin7 environment has been more support intensive than I would like both on my desktop and notebook. But that maybe the result of a dreaded root kit that hit my desktop. Tough calculating the residual impact of that even though I have rebuilt both systems with disk reformats and operating system reinstalls. I will look hard at your solutions. Thanks again for your always useful input

Intel has clout and I predict we will see Thunderbolt on most PCs in the future. This doesn't help you now though. Apple was an early adopter and it has taken AGES for TB peripherals to come out but they are starting to be seen more and more.

[Steve Uffman]

Intel has clout and I predict we will see Thunderbolt on most PCs in the future. This doesn't help you now though. Apple was an early adopter and it has taken AGES for TB peripherals to come out but they are starting to be seen more and more.

Thanks,

agree that Thunderbolt will show up on PC and a few manufacturers are announcing new units with it...HP has said no to the idea....Dell has been somewhat silent....Do think it will happen.....Problem is the way Thunderbolt works, it likely will be a motherboard issue rather than a quick and easy as the USB 3.0....So that probably means new box to get it.....
So, I am finally giving serious consideration to going away from the Win platforms and thinking Apple.....3 system rebuilds in a 12 month period on two machines is too much.....rootkit problem may have contributed but tired of fighting it....the dropping drives, windows updates periodically not working and yada, yada.....

will look at the Seagate drives as well....and completely agree on the SSD for operating systems...almost did that on my notebook since I use the passport external drive when I am in the field.


And on your intermediate files, I have seen 60MB and 3x that on intermediate files....that eats a ton of space...and I can usually redo the image from RAW pretty quickly as I don't get to elaborate

Steve

http://uffmannatureimages.net/

[Roger Clark]

Steve,

Whatever system one chooses (mac, windows, linux), fast I/O for getting data in and out is very important in my opinion. This can be especially important on a photo trip like a safari. Thunderbolt is spec'd at 10 gigabits/second, USB 3 is 5 gbits/second, and USB2 much slower. I have seen real throughput of 110 megabytes/second (mbytes/s) with USB3, but on some machines only 30 mbytes/s on others, about the same speed a USB2 on a fast I/O machine. It would be interesting to hear real world throughput of thunderbolt. On windows laptops, I have seen machines with slow USB3 of only 9 mbytes/s. I've also observed 110 mbytes/s from USB3 disks across a gigabit ethernet network to a gigabit iSCSI disk (linux to linux).

Roger

[Roger Clark]

And on your intermediate files, I have seen 60MB and 3x that on intermediate files....that eats a ton of space..

Hi Steve,
I sometimes save intermediate files, but when they have layers, I save a psd (or photoshop's large file format), as it compresses quite well. When I'm making mosaics, I save the mosaic from ptgui and both layers and blended mosaic and file size is often several gigabytes.

Roger

[John Chardine]

Steve,

Whatever system one chooses (mac, windows, linux), fast I/O for getting data in and out is very important in my opinion. This can be especially important on a photo trip like a safari. Thunderbolt is spec'd at 10 gigabits/second, USB 3 is 5 gbits/second, and USB2 much slower. I have seen real throughput of 110 megabytes/second (mbytes/s) with USB3, but on some machines only 30 mbytes/s on others, about the same speed a USB2 on a fast I/O machine. It would be interesting to hear real world throughput of thunderbolt. On windows laptops, I have seen machines with slow USB3 of only 9 mbytes/s. I've also observed 110 mbytes/s from USB3 disks across a gigabit ethernet network to a gigabit iSCSI disk (linux to linux).

Roger

Agree Roger. The jump from USB2 to Thunderbolt has been huge for me. I never used USB3 but would likely have been happy with it.

I tested my system using Blackmagic Disk Speed Test running on a Macbook Air 1.7 GHz Intel Core i5 with 4GB of internal memory, Mountain Lion 10.8.1. I get about 130-140 mbytes/s read and writes. I am running a 27" Thunderbolt monitor off the same TB port in a daisy chain. The monitor is first and the drive is second. A third drive on the chain tests out at about 120 mbytes/sec read and write, so slightly slower than the first. Removal of the TB monitor from the chain does not appear to change the throughput.

Thunderbolt is capable of a lot more throughput as seen here after the presenter sets up a RAID system (note how easy that is to do by the way with included Disk Utility).

http://www.youtube.com/watch?v=QRGehelveIg

By the way, I'm really looking forward to a Thunderbolt CF card reader!

[Greg Basco]

Hi Steve,
I sometimes save intermediate files, but when they have layers, I save a psd (or photoshop's large file format), as it compresses quite well. When I'm making mosaics, I save the mosaic from ptgui and both layers and blended mosaic and file size is often several gigabytes.

Roger

Hi, Roger. Steve, sorry to jump in, but I just wanted to ask Roger if it is indeed the case that a 16-bit PSD file will be smaller than a 16-bit TIFF and if, in the latter case, LZW compression really does much with 16-bit. I appreciate your thoughts, Roger.

Cheers,
Greg Basco

[John Chardine]

Hi Greg- I just did a small test of a full-res, 16-bit unprocessed file from the 1DIV:

Saved as a .psd: 96mb
Saved as a .tif no compression: 96mb

Now comes the interesting part- just duped the BG layer and tried again:

Saved as .psd: 254mb
Saved as .tif uncompressed: 320mb

Adding a layer roughly triples the file size and the .tif is much bigger than than the .psd.

[Steve Uffman]

Enjoying the discussion and its been most helpful. The scenario that John describes above has been my experience even if you optimize the compression. Since I hopefully am getting more skilled with both my image taking and PS work, saving the "Master" file with layers intact seems to be the best practice as separately discussed with Steve Kaluski... Chas is out of the country, but my notes from one of his STLs suggests he does the same thing....That is why my average "cumulative" image footprint on my better images is getting huge. And having been 20 days in Africa, the problem is such that I cannot yet move my images over to my desktop.

I have been doing a significant amount of research on this and if one stays PC, a highly respected photographer and technical guru recommended going with 3tb internal drives where the throughput of course trumps external via thunderbolt or USB 3.0 ..plus an SSD for the boot drive. At my old age, I would like to break this perpetual 2 yr upgrade cycle that I seem to be in (maybe go to 4 yrs?). If I shift over to the dark side (I predate Microsoft and Intel so I grew up with them and have some loyalty built up over generations), John's apple solution looks very, very strong. Am going to investigate Linux as Roger suggests as well....was never a Unix guy so have some old biases to get over...but the Win7Pro reliability which is either inherent or a carryover from the dreaded rootkit that invaded my machines awhile back, is way too much maintenance for my temperament these days.

[Steve Uffman]

Hi, Roger. Steve, sorry to jump in, but I just wanted to ask Roger if it is indeed the case that a 16-bit PSD file will be smaller than a 16-bit TIFF and if, in the latter case, LZW compression really does much with 16-bit. I appreciate your thoughts, Roger.

Cheers,
Greg Basco

I do the LZW compression which makes a big difference as John describes in a subsequent thread...but the footprint is still huge....

[Steve Uffman]

Steve,

Having used external hard drives for years I found them very frustrating, so I bought an external server to store everything I got a QNAP with 4 x 2TB Hitachi 2TB disks, and touch wood it has worked so far, although Windows sometimes cannot recognise it is there

The only problem seems to be the speed of transfer, but that is probably down to my "African" router

Also meant to point out the value of Ken's suggestion here on multiple external drives in an enclosure which I think is fan cooled ( if I looked at the right thing). Using multiple WD mybooks as external drives as I have which are not fan cooled can be an issue as I have studied further....Actually, I know this from my previous worklife but overlooked this as I incrementally added storage over time.. Certainly they are all in AC and never feel that warm but who knows what heat multiple drives can exchange when they are working hard. Just want to eliminate all potential culprits.


Ken, your enclosure is fan cooled, correct?

[KevinStohlgren]

I am also currently looking into an image storage/data backup solution. Unfortunately a lot of this conversation is going over my head.

I currently just have just one 300 gb Western Digital drive that I store files on, but if that goes, I am screwed. So I am looking for recommendations for a solution to this problem. I am a poor graduate student who moves around a lot and has little tech knowledge, so I am looking for something small, cheap, and easy to use.

Would a pair redundant drives (where one could be stored off-site) suffice my limited needs (this is what I am currently thinking)? Currently I am just dragging files from my desktop over to my external HD, but if I had two drives, how do I ensure that both have identical data? And if I want to work on a file that has been stored on the external drives, how do I ensure that both drives will be updated? Do you all recommend using a data backup program, or somehow do it manually? If a program, how do they work (a very brief explanation is fine)?

[John Chardine]

Hi Kevin- IMO you can't beat the small Western Digital Passport drives. They have a nice 1GB version that connects via USB2 and maybe 3 by now too. I would get two or three of them and use two for backups- one on-site and one off-site and rotate between each one. This is assuming the 1GB will be sufficient for now.

Regarding backups you really need to use a backup app of some sort so that only files that have changed since the last backup are copied across. This makes the backup job efficient. Not sure what platform you are on but if it's Windows I will let someone else chime in on app options. On Mac, Time Machine (included with system software) is fabulous.

[Greg Basco]

Hi, Kevin. I think John gave a good suggestion with the passports -- relatively cheap, reliable (I've never had a problem with mine), and easy to move around/store offsite. For PC, I use the free SyncBack program. It's quite easy to set up and will make sure that each drive contains the most up to date versions of the target folder(s) on your computer.

Cheers,
Greg

[KevinStohlgren]

Thanks guys. I am currently shooting with the Sony A77 which has really large files, so 1gb drives are not going to be enough. But memory is cheap enough that the size of the drives aren't what is really concerning me, I can figure me needs out there.

My question now is, with the software you are recommending (I am using Windows 7), do I have to keep the files on my computer? I would prefer to not have to keep all of my pictures on my desktop to save space there. To do this would I then have to get a three hard drive setup where one is used as my primary storage and then the other two serve as backups which I switch out regularly (keeping one off site)? Or is there still a way to do this with just two hard drives?

[Roger Clark]

Thanks guys. I am currently shooting with the Sony A77 which has really large files, so 1gb drives are not going to be enough. But memory is cheap enough that the size of the drives aren't what is really concerning me, I can figure me needs out there.

My question now is, with the software you are recommending (I am using Windows 7), do I have to keep the files on my computer? I would prefer to not have to keep all of my pictures on my desktop to save space there. To do this would I then have to get a three hard drive setup where one is used as my primary storage and then the other two serve as backups which I switch out regularly (keeping one off site)? Or is there still a way to do this with just two hard drives?

Kevin,
How much data do you have? You mentioned above that has a WD 300 GB drive. That does not sound like a lot of data. Most computers have room for at least 2 drives. So you might be able to add a second drive (e.g. 3 TBytes) and have plenty of space.

Roger

[Greg Basco]

Kevin, you wouldn't want to use an external as the primary location for your images IMO because of speed issues. As Roger mentioned, a second internal hard drive is a good option. That's what I have in my PC -- a second internal 2 TB drive.

Cheers,
Greg

[KevinStohlgren]

Thanks again, guys.

Yes, Roger. 300gb is not enough. I currently have 200gb of photos sitting on my computer right now. My computer is a HP Pavilion dv7 with a 700gb hard drive. A quick search shows that I can add a second hard drive (1 tb max?). How do I go about finding one that will fit and is it simple enough to install myself (I am not completely useless)? Does this serve as a backup that would regularly backup the files I want it to and would it be protected from catastrophic events such as lightning strike (I did lose a hard drive on a desktop computer that way about 10 yrs ago)? Could I then get away with only one external backup that was updated regularly in case my computer was dropped or stolen, or would you all still recommend two?

[Jim Neiger]

I have an internal RAID 0 (2 drives of 1TB each - 1TB space available for storage) I use the internal RAID for storing recent images (about 3 months worth) and as a work area for any images I'm currently working with. I have about 15tb of external drives. The drives vary in size from 1TB to 2TB. The external drives are used to store older images arranged by date. Each external drive has another drive that is an exact copy. All of my external drives are used as read only once I have filled them up with images. One copy is attached to my system. It's copy is stored off site. If I want to use an image that is on an external drive, I copy it to my internal working RAID for processing. Each time I run out of space I buy another pair of external drives. I plan to add a third copy of each drive for redundancy, but I haven't got around to that yet. As bigger faster drives become available, I buy them and copy data from the old drives to the new drives. The old drives are sitting on a shelf unused. Someday I will either sell or throw out the old drives, for now they are sort of a third backup of some of my data.


When I cull images I do two passes thru the images. The first pass I try to do while the images are still on the flash cards. I select the keepers and download them. At some future date I go back thru the images and cull them down further. I am doing this chronologically and I am way behind. Right now I'm up to July of 2006. For me this has been a problem of too many keepers. I am really bad at culling images. If it's a sharp and pleasing image, I keep it on the fist pass which I do very quickly spending about 1 second or less viewing each image. The second pass takes much longer. I think a better method is to cull ruthlessly on the first pass keeping only those images that are really excellant. Artie is a master of this and as ruthless as they come. I have seen him start with a couple of thousand images and end up with less than 10 after culling. I've tried to do this, but I have a hard time throwing away really good images. Now I'm paying the price for that bacuse I have too many images and not enough time to process them all or even go thru and cull them further. Maybe in another 5 or 10 years I'll learn to be as ruthless as Artie when it comes to culling images.

[Roger Clark]

Thanks again, guys.

Yes, Roger. 300gb is not enough. I currently have 200gb of photos sitting on my computer right now. My computer is a HP Pavilion dv7 with a 700gb hard drive. A quick search shows that I can add a second hard drive (1 tb max?). How do I go about finding one that will fit and is it simple enough to install myself (I am not completely useless)? Does this serve as a backup that would regularly backup the files I want it to and would it be protected from catastrophic events such as lightning strike (I did lose a hard drive on a desktop computer that way about 10 yrs ago)? Could I then get away with only one external backup that was updated regularly in case my computer was dropped or stolen, or would you all still recommend two?

Kevin,
You probably can add a second hard drive, but check you manual to be sure. With windows 7 you should be able to use 3 TB disks, but some people seem to have issues with windows. 2TB should be no problem, unless the hardware is pretty old. As you already have a 700 GB system drive, 1TB should be fine too. If you only have 200 GB of photos, upgrading to 1 TB data drive would give lots of room to expand. it is pretty simple to add a drive, but it would help for the first time to have a friend do it and show you. Your manual will tell what kind of drive (or look up online). If a newer system, it will be SATA.

The more backups you have, the safer you are. I believe it is especially important to have an off site backup (protects against disasters like fire/flodd and robbery). I recommend a minimum of two backups stored offline. 1TB USB drives are pretty cheap (been there as a poor graduate student).

I run 3 USB backup sets, rotating them, and one online backup on another server in my house. Plus I back up my original CF cards to a portable drive, so 5 backups of my original data, 4 of original + processed data.

Roger

[Steve Uffman]

Backup, backup, backup is great advice for sure....and offsite is very important...A friend lost all of their images during a hurricane flood because he did not take the extra backup off site

Getting close on my new machine specs which looks like a custom build so I don't have to deal with this every two years....(although I am sure I am overly optimistic)

[KevinStohlgren]

Alright, so here are my thoughts. I either pick up a second internal drive (my computer can hold a 1 tb SATA drive) and an external drive that can be kept off site but backed up regularly. Or I can just pick up two externals and rotate them. With either option, when I eventually run out of space I can pick up another pair of external drives to use as storage such as Jim does.

So of those two methods (a second internal + 1 external vs. 2 externals), is there any major advantage of one over the other? Obviously a second internal would be faster but would the difference be significant to someone with limited computing needs? I see the down side of the second internal drive being that when I eventually upgrade to a new computer, that second drive would no longer be able to be used (although I know the sell cases for them so you can use them as externals). Price difference seems to be negligible.

[John Chardine]

It's important to note that RAID should not be used as a backup strategy. If a problem occurs you can't go back in time and recover an earlier version of a file or a whole disk- this is the essence of backing up. With RAID 0, a corrupted file or deleted file state is just blindly copied to the other drive with no option to go back and undo. An effective backup system allows this. RAID protects you from outright disk crashes but often disks fail slowly, and you just end up making a duplicate of a failing drive.

[Roger Clark]

It's important to note that RAID should not be used as a backup strategy. If a problem occurs you can't go back in time and recover an earlier version of a file or a whole disk- this is the essence of backing up. With RAID 0, a corrupted file or deleted file state is just blindly copied to the other drive with no option to go back and undo. An effective backup system allows this. RAID protects you from outright disk crashes but often disks fail slowly, and you just end up making a duplicate of a failing drive.

Hi John,
I agree with the above, but feel there needs to be some clarification. Perhaps saying a single RAID alone is not a good backup strategy. For example, at work I have to back up over a hundred terabytes of data (and growing) and we do it by backing up two copies to off site raid systems. And even raid can fail. I've had 3 raid failures in my career that destroyed data: Twice on a Friday after everyone went home, fans failed and the drives in the raid enclosure cooked the drives over the weekend and all 16 drives in the raid box failed. The most recent incident was a raid controller failed in such a way that it lost its addressing so it started writing random data to random locations on random raid systems! Destroyed around 50 terabytes of data in a short period on a Monday morning before we shut it down. In all cases, data were restored from backup raid systems and nothing was lost. We now include offline archive backup on 4 terabyte USB drives.

The key is multiple backups, at least some off site and offline.

Roger

[Roger Clark]

Alright, so here are my thoughts. I either pick up a second internal drive (my computer can hold a 1 tb SATA drive) and an external drive that can be kept off site but backed up regularly. Or I can just pick up two externals and rotate them. With either option, when I eventually run out of space I can pick up another pair of external drives to use as storage such as Jim does.

So of those two methods (a second internal + 1 external vs. 2 externals), is there any major advantage of one over the other? Obviously a second internal would be faster but would the difference be significant to someone with limited computing needs? I see the down side of the second internal drive being that when I eventually upgrade to a new computer, that second drive would no longer be able to be used (although I know the sell cases for them so you can use them as externals). Price difference seems to be negligible.

Kevin,
As you have only 200 GB of photos and a 700 GB internal drive now, which implies you still have room on your system, I would recommend the 2 external drives as backup. I suggest buying at least 2 TByte drives for backup as it will give you room to grow. You need not go all out at one time. For example, buy one drive now and get your data backed up. When money allows, buy a second drive and start rotating. Keep one in the same location as the computer so you can back up whenever you add significant new data, and keep the second off site. As money allows, then get a second internal hard drive. In a year or so, add a third USB drive so you have a 3-drive backup into your rotating backup strategy.

In all of these strategies of rotating backups, it is possible for a file to be corrupted (e.g. written over with other data), so I feel it is also important to make a "frozen" archive backup periodically that you never touch again. I do that by saving my internal drives. For example, I currently have two 2-TByte internal data drives that are getting full. I'll upgrade them to 3 (or 4) TByte drives in the next few weeks. When I pull the old drives, they will go on a shelf (perhaps off site) as my archive.

Off site ideas: if you have an office at work where you can lock up the drives, great. If you don't have an off site office, one could rent a safe deposit box, or perhaps start an exchange with friends or family. But be sure they are far enough away to no be hit by the same disaster. For example, the family member a block away and you are both in a hurricane/flood zone is not a good off-site strategy.

Roger

[Steve Uffman]

Amen to Roger about hurricanes and flood...During Katrina, some of even the big corporations were stupid in that their offsites were in the same city as their data center...shocking that they would do that but I know first hand of folks that did that in New Orleans and Houston....For us, our data center had the offsite backup in the North east US. Also very much agree with Roger's suggested strategy...It is how I started out ...and like Roger I am in the same situation so the old drive will go on shelf as a snapshot and the larger new ones will replace it....

[KevinStohlgren]

Thanks once again for all the suggestions guys. This sounds like a very good solution, and it will be comforting to get all my photos (and probably more importantly my thesis work) backed up.

[Roger Clark]

(and probably more importantly my thesis work) backed up.

Kevin,
For something that important, I would suggest more backups, including to different media. For example, also buy USB memory sticks and make an archive backup of your thesis right away, then start a couple of memory stick rotating the backup of your thesis. When I'm working on an important project where I have a lot of time invested, I have at least 5 backups and at least two different media (e.g. Hard Drive and USB memory stick). And I increment versions and keep backup of all versions. Call me paranoid, but I have never lost any data and I have seen others lose theirs (including accidentally deleting their only copy)

Roger

[Steve Uffman]

FWIW,

Below are the specs which I am either going to build myself or have someone build for me.
i7-3920XM Processor
256GB SSD SATA 3 (6GB/s)
32GB of DDR3-1600 (upgradable to 64gb
High end nVidia or ATI graphics card
Multiple 3TB Seagate Barracuda drives (expandable to 4 drives)
1000W Power supply
5 usb 3.0 ports and other connectivity options

Still determining the backup equipment but I want an air cooled chassis, so studying several with removable drives.

[Roger Clark]

FWIW,

Below are the specs which I am either going to build myself or have someone build for me.
i7-3920XM Processor
256GB SSD SATA 3 (6GB/s)
32GB of DDR3-1600 (upgradable to 64gb
High end nVidia or ATI graphics card
Multiple 3TB Seagate Barracuda drives (expandable to 4 drives)
1000W Power supply
5 usb 3.0 ports and other connectivity options

Still determining the backup equipment but I want an air cooled chassis, so studying several with removable drives.

Looks impressive, Steve.

I've been using Coolermaster chassis with 6 to 7-inch fans. The larger fans are much quieter. The cases that hold 4 or so drives with big fans are quite large though. The ones I get can hold 6 hard drives.

What motherboard? Impressive to get up to 64GB.

Roger