Raw Storage Organization + workflow

[Jay Gould]

What various systems for retrieving original RAW images have you found successful?

I am going to be travelling for seven months. If I shot just for discussion 500/day and reduced to 50/day, that is still 10,000+ images! :eek: Help!! :o

Master folder describing trip, or year date?

Sub-folders by date or by location description and further sub-folders by date or description?

Even though the images are keyworded, that only works when the images are opened in a program that recognizes the keywords. What about retreival of images that are archived? How are you doing that? Date doesn't seem to give enough direction to a particular location unless you have an index to refer to if organized by date.

I am not an image saver; hopefully after downloading the initial images into an appropriate folder (and making a backup), and after applying Breeze Browser, I will only have 10 - 15% of original shoot.

The remaining 10% will then be opened in Lightroom and batch renamed, and duplicated replacing the original 100% folder.

After renaming and duplicating, I now have two or more duplicate folders; each folder contains the 50 images plus the xmp file. Next step would be whatever editing I do in LR.

After applying keywording etc to that original 50/day (whenever that finally happens), and maintaining the images as CR2 which means an .XMP sidecar file for every image, I assume it then becomes necessary to duplicate that folder with the 50 images and 50 xmp files to replace the unedited CR2 files/folder that does not have keywords etc.

Since all editing done in LR is nondestructive of the original RAW file and is maintained in the xmp file, if you did not change the file name, couldn't you simply duplicate all of the xmp files for a particular folder and add those xmp files to your backup folders?

Finally, from LR I would go to PS-CS4.

I am shooting in Adobe RBG and my default workspace is ProPhotoRGB for the largest possible colorspace.

Before doing any work in PS, are you saving the particular image as a PSD or a TIFF? I know there has been a lot of discussion about this issue and I am still not clear why some folks are saving as a TIFF which is so much larger than a PSD.

Thanks for all of your assistance; I can wait to start posting images from the trip to share with you.

:D:D

[Roger Clark]

Hi Jay,

Boy, I would have a problem with that scenario. The effort to delete 90% of my images while on a trip would be very difficult. Mainly because of the time involved. While on a trip, I want to have fun and enjoy it and not spend all my spare time on a computer deleting images. I've also found that images that I reviewed quickly on my laptop and thought not sharp turned out spectacular when I got home and viewed them on my large calibrated monitor. So I only delete the really bad out of focus/blown highlight etc images. On my January trip to Tanzania I took about 4500 images in 12 days or 375/day (down from 8000 the previous trip, or 667/day; not because the photo ops were worse--more like I controlled what I took better). Of those 4500, 90+% are in super focus, good exposure, making it hard to make the decision as to which to delete in the field. Much spare time was simply spent backing up the cards, organizing the images, and making sure the cameras were working properly (Oh, and the ooing and ahing over the images).

I organize my images into directories with the name equal to the place, date and time. e.g.
africa.2009
ndutu.2009.22.01.am
ngorongoro.2009.22.01.pm

When action is higher and I have less time, I simply dump my cards. I number each card in case a card goes flaky, I can see it later and identify it. e.g. r19, r20, r21, r22, r23, r24, so I make directories:

africa.2009
a.r23
b.r19
c.r36
I then move images into the place+date directories when I get time. The a, b,c is the order I fill the cards.

Laptop hard drives are so cheap I've now decided I'll dump my cards to 500 GB (or larger) hard drives and usually not worry about deletion unless I have a lot of time. I'm archiving the hard drives at least for a while and not erasing them. I copy the hard drive to my desktop computer after I return from my trip and then delete on my desktop. The original drive contains all the images in case I delete something I want. (Of course I do have multiple drives for backup.)

Now this becomes more difficult for your extended trip. But 640 GByte laptop drives are out and 1 TB drives have been announced (might be able to get them now). A 500 GB drive will hold about 16,000 5D II raw plus jpeg images. Don't forget your 5DII can also do spectacular HD video, and with your telephoto, think Planet Earth (the BBC series) quality is possible. So take lots of storage!

I would think about taking around six 1-terabyte hard drives. (or 10 640s). Could you mail them as they fill to
a relative back home and keep a copy with you as additional backup?

I find it would be difficult for me to take 500 images/day for ~200 days. 12 days in Tanzania obtaining 4000 to 8000 images is exhausting, e.g. the schedule of getting up at 5:30 am each day plus staying up after a days shoot backing up cards. At least on the equator the sun gets high and the light harsh, so we stop photographing. In Antarctica, the sun will always be low and the sun never sets so you can photograph 24 hours per day. So how are you going to go without sleep for 200 days??! ;) Boy, what a problem! Maybe 500 images per day is low. Then when will you find the time to edit? (I'm not trying to scare you--I'm jealous!);););)

Fortunately, computer technology is moving fast enough that it is plausible to take the disks to backup all those images.

Roger

[Mark Fuge]

Jay, the biggest problem you have is storage during the trip.

I personally agree with Roger. Deleting images on the fly is risky at best. There is no way to really tell the quality of the image the day you shoot it. Mainly because you are still emotionally tied to the events of the day.

I would try to find a way to download them for storage and keep all images that have anything viewable. When you get home, take your second vacation to sort through them.

When I was shooting a lot of slides, I often found that 2 - 3 weeks after the exposure of the image the image was totally different to me mentally. Images that on first viewing were defininate throw aways turned out to be treasured images when the emotiions left and I could look at the "art" of the image.

On my one month tour of Hawaii, I had about 70 rolls of 36 exposures when I got back. I would have had more but I spent two of those weeks with family while there. There is no way I would have wanted to edit them on the run.

My current filing system consists of Topic Folder - Sub Topic Folder - Date Folder - File Item Date - Location - Title - Image #. It looks more complicated than it is.

Example

Topic Folder - Florida
Sub Topic Folder - Merritt Island NWR
File (Shooting) Date Folder - 20090615
File ID
Date - Location - Title - Image#
20090615 - MINWR - Egret on Point - IMG0051

This way the Date is the first thing in the "title" of the file item and is easily sorted. When I print an image, I put the File ID on the back with the title and copyright.

Good Luck and buy a lot of Memory Discs!

You may want to go to the old practice of using memory discs and sending them home so they are there when you return. I use to do this with film that I had developed and shipped right to my house. The first batch usually beat me home, in your case that wouldn't be a problem. But if something happened on the trip, most of your images would still be safe at home. Send them securely, such as FEDEX, etc. and feel secure as well.

Good Luck!

[Jay Gould]

Hi Roger and Mark, thanks for your thoughts.

To begin with, when I think of the number of images I will shoot over seven months, it is daunting. You guys are experts; I am a beginner. I have no doubt that I will have my finger on the trigger longer than either of you when doing bursts - at 8fps with the new 7D. Clearly that will quickly add up to thousands of images.

While I will not be shooting at that level every day, when I look at the numbers you two shoot (375/day in Tanzania; 180/day in Hawaii) and if I only shot on average for the whole trip 400/day, that would mean facing 80,000 images :eek: - an overwhelming amount.

So, my goal is to only face between 10 - 15,000 images.

I am intending to use Breeze Browser (thank you Artie) and to be ruthless; I have no choice!

In reality while Roger and I assume Mark too find that 90+% are in super focus/good exposure, as a beginner I do not expect to have that problem you describe. Hopefully as the trip progresses I will increase my "think about keeping" rate.

I am not a saver - never have been - never will be. I suppose it comes from living in small spaces (boats and caravans) since 1987. At the end of the day, if an image isn't worth printing or posting on a website I hope to create starting with my SA images, it isn't worth saving. Having an image in my "collection" isn't important.

In terms of the media upon which I am performing my review, we start with the 3" screen on the HyperSpace if sitting around in the field; otherwise, I will be doing all reviews and whatever PP on a 13" Dell. At home everything is done on a 17" Dell. No fancy large monitors; no room in the caravan.

Regarding storage, I will start by dumping the cards into the 500gb HyperSpace, and saving the cards until I can quickly look at the results and make sure they transferred properly. Then I will transfer from the HyperSpace to my computer and to one of two 500gb LaCie hard drives. As a I cull I will delete from the various storage devices. Ultimately, I would have duplicate copies on the two LaCie drives, and whatever I am working on will be on the computer hard drive.

If a 500gb will hold 16,000 images, then my goal will be to have no more than 16,000 images to bring home.

When you travel for the lengths of time we travel, we believe in the every other day mode of sightseeing. We are very into lots of down time; long lunches with a bottle of wine and a good book or in my case a 13" computer from which I can be deleting images. Sometimes when the image quantity gets ahead of us, we schedule an image deletion day. :D

In terms of filing, I like the idea of starting with location/date and then going to subfolders, e.g., South America 2009, Peru 2009, etc.

Again, thanks for taking the time to read and comment.

[John Chardine]

You will need to use some sort of Digital Asset Management (DAM) software (AKA image database). Examples are Adobe Bridge, LR, iView Media Pro (now called Expression Media by Microsoft). I use the latter and find it excellent. As date/ time is stored as EXIF data with each image it is redundant to impose a folder structure on your hard disk that involves date. Your DAM software of choice will allow you to sort images by date/time or pick out a specific date or date range. After I catalogue my images I attach them to a location in my DAM software, or add a new location as needed. I keep a separate hand-written or computer diary of my travels.

This way, images can be dumped in a single folder on your hard disk. This is much simpler and makes backing up simpler too. KEEP IT SIMPLE is an adage I would highly recommend for this.

[Cliff Beittel]

Jay,

A good book to read, if you have time, is Peter Krogh's The DAM Book, Second Edition. Seemingly little things like file names and folder structure make big differences down the road. There's lots of good info in the book on how to make those decisions.

I've had very good luck with two older HyperSpace units (no screens, just storage). I've had very poor experience with Lacies (I bought a pair of Lacie Ruggeds, one stopped working in less than a year of very easy use). I've also read about lots of other folks' problems with Lacies, so personally I would never count on a Lacie drive getting me through six or nine months of shooting. No matter what the brand, if I had no way to buy replacements on route, I'd want to start with four drives to be sure of having two still working at the end.

[Jay Gould]

You will need to use some sort of Digital Asset Management (DAM) software (AKA image database). Examples are Adobe Bridge, LR, iView Media Pro (now called Expression Media by Microsoft). I use the latter and find it excellent. As date/ time is stored as EXIF data with each image it is redundant to impose a folder structure on your hard disk that involves date. Your DAM software of choice will allow you to sort images by date/time or pick out a specific date or date range. After I catalogue my images I attach them to a location in my DAM software, or add a new location as needed. I keep a separate hand-written or computer diary of my travels.

This way, images can be dumped in a single folder on your hard disk. This is much simpler and makes backing up simpler too. KEEP IT SIMPLE is an adage I would highly recommend for this.

Hi John, I use Bridge and LR. Regarding the data structure, starting with date rather than place to me seems like asking for trouble because when you look at a folder structure and all you see are dates - I am not going to remember where I was on a particular date.

As I indicated earlier I am not a saver. I would hope that no matter how many I shot during the seven months, that would be reduced to less than a thousand :eek: images by the time I finish culling and editing.

I think it was Roger in some thread that said he had over 150,000 images - I cannot image saving that many images, and I suspect that if I did then someday my kids would dump then when I am long gone; I will do it for them!

[Jay Gould]

Jay,

A good book to read, if you have time, is Peter Krogh's The DAM Book, Second Edition. Seemingly little things like file names and folder structure make big differences down the road. There's lots of good info in the book on how to make those decisions.

I've had very good luck with two older HyperSpace units (no screens, just storage). I've had very poor experience with Lacies (I bought a pair of Lacie Ruggeds, one stopped working in less than a year of very easy use). I've also read about lots of other folks' problems with Lacies, so personally I would never count on a Lacie drive getting me through six or nine months of shooting. No matter what the brand, if I had no way to buy replacements on route, I'd want to start with four drives to be sure of having two still working at the end.

Thanks Cliff; I will see if it is in the local library.

I have been using two Ruggeds for quite some time - so far no problem. I will always keep the two with mirror images. If one starts to play up I will buy another hard drive during the trip. We are going to be in and out of major cities in SA; I will find something.

I have always wondered who made the "best" hard drives. In the space of a couple of months I went through four different 1T hard drives; now using the LaCie Quadra.

[John Chardine]

Hi John, I use Bridge and LR. Regarding the data structure, starting with date rather than place to me seems like asking for trouble because when you look at a folder structure and all you see are dates - I am not going to remember where I was on a particular date.

As I indicated earlier I am not a saver. I would hope that no matter how many I shot during the seven months, that would be reduced to less than a thousand :eek: images by the time I finish culling and editing.

I think it was Roger in some thread that said he had over 150,000 images - I cannot image saving that many images, and I suspect that if I did then someday my kids would dump then when I am long gone; I will do it for them!

The point is Jay that in the system I use there is no data structure to speak of on the hard disk. The EXIF data for each image contains date and so there is no point in making folders for each day you shoot. I can see the logic of making folders for locations but that's why I said I enter place names into the database either right after shooting when it's fresh in my mind or later on when I might need to refer to my diary. I recommend in the strongest terms that you keep a diary so that you CAN remember where you were on a particular date.

[Cliff Beittel]

. . . I have always wondered who made the "best" hard drives. In the space of a couple of months I went through four different 1T hard drives; now using the LaCie Quadra.
Jay,

It changes over time. Seagate once had a great reputation, then more recently they had problems. From what I've read, Western, Hitachi, and Samsung are currently the most reliable. I've personally had good luck with Seagate, Western, and Samsung in my desktop computers and main storage enclosures. I believe my Lacie Ruggeds had Samsungs inside, and I suspect it wasn't the drive itself that caused me trouble, but the USB connection in the Lacie case. But one of the problems of buying enclosures from companies like Lacie is that they don't make their own drives, so you probably won't know what's inside when you purchase. All drives, even the best ones, fail eventually, and some will fail prematurely, which is why I'd want at least three copies of my images (which you would have with your two Lacies as long as there's a third copy on your computer).

I assume solid state drives will provide much better reliability--akin to what you get with a good compact flash card. They are still very expensive at the moment in sizes that would be useful for storage.

[LouBuonomo]

and I would not rely on 1 hyperdrive.. I would have 2 or a Netbook or something with an external. Packed in differnet luggage.

[Roger Clark]

I recommend in the strongest terms that you keep a diary so that you CAN remember where you were on a particular date.

John,
Diaries can get separated from you data. That is why a simple directory structure, with place and time is good, e.g.
sandiego.2009.01.23
yellowstone.2008.06.02
mt_rainier.2009.07.15

Throwing all your files into one directory is dangerous. As the number of files grows, it takes more time to search them. If a directory becomes corrupted you have major problems. (e.g. google: corrupted directory).
Backup is more complicated too. Management of smaller numbers of files per directory is easier, including listing, searching, and backup.

Roger

[Jay Gould]

and I would not rely on 1 hyperdrive.. I would have 2 or a Netbook or something with an external. Packed in differnet luggage.

Hi Lou, I am going to have one HyperSpace and two LaCie Ruggeds. I will initially, in the field, download each card into the HyperSpace as they are full, and I will keep the cards until I return to "base" at the end of the day. Then, using the cards I will download to both Ruggeds, and to my 13" laptop.

In theory there would only be 500+- images which I would quickly go through using BB. Hopefully, the first "dump" will be 80-90% of the images. I will then transfer that 10-20% to a permanent folder on each of the Ruggeds and delete the original 500 from each Rugged. And life goes on!

It will be fun next May or sooner depending upon internet time to share what actually happens in the field, and of course to share some of the "keepers".

[Dan Brown]

Hi Jay. Sounds like you have a good plan here! I haven't read every post here but I will add that the only thing that might foul you up is if you have shooting down time and you start working on images with PS and saving layers (photomerges, HDR'S, OOTB stuff, etc) you may start gobbling up storage space! Sound like a great trip for you coming up!

[Jay Gould]

Hi Jay. Sounds like you have a good plan here! I haven't read every post here but I will add that the only thing that might foul you up is if you have shooting down time and you start working on images with PS and saving layers (photomerges, HDR'S, OOTB stuff, etc) you may start gobbling up storage space! Sound like a great trip for you coming up!

Dan, between you and the other posters you guys are making me nervous about storage! I am seriously considering buying two Toshiba 500gb HDs. They are only $92/free shipping; I have always relied upon Toshiba products.

The storage thing again raises for me the question of PSD compared to TIFF. TIFF files are so much larger; I have never received an adequate answer why those that use PSD which is an Adobe proprietary are not concerned that it might not be viable in the future. I have also never received what I consider to be an adequate answer why those that use TIFF which is megabytes larger are so afraid that PSD will be gone in the future when Photoshop is here to stay at least in my lifetime.

Is there ANYTHING that can't be done with a PSD that can be done with a TIFF?

I know, I am hijacking my own thread.

[Sabyasachi Patra]

Dan,
I rename my files with the year, month, day and the unique number of the file. So from the file I can know the date it was clicked. I save the files date wise in folders and these go into a parent folder name according to location. I keyword them while importing into lightroom. I save my RAWs and today I can go back and reprocess again. In the initial days my tigers used to look more colourful :D:D Now I can correct that.

I would say don't delete in the field. Carry more hard drives than clothes. Animals won't mind if your clothes are soiled. :D

Cheers,
Sabyasachi