Results 1 to 17 of 17

Thread: suggestions for External Hard Drive config for network that I am filling up with images very quickly

  1. #1
    BPN Member Steve Uffman's Avatar
    Join Date
    Jan 2011
    Location
    Baton Rouge, LA
    Posts
    586
    Threads
    77
    Thank You Posts

    Default suggestions for External Hard Drive config for network that I am filling up with images very quickly

    time to refresh my drive systems and backup for my network that is primarily servicing my photo hobby.

    Interested in thoughts as I am about out of space on my desktop with internal 1tb and 1tb wd external drive that I use backup my images and basic outlook, web and stuff.....

    my network has two laptops and desktop ...I had some legacy HP external drives (250MB and 500MB) that I have been backing up the individual laptops ....

    At the image rate I am now taking and with the travel scheduled in the next 6 months, I will blow through my available storage and backup

    I use LR3 and CS5

    laptops are on wireless connection....all backup devices are direct connect

    I don't need to reinvent the wheel..surely the wizards have thoughts on how best to do this?

    thanks in advance for your suggestions

  2. #2
    BPN Viewer
    Join Date
    Jan 2008
    Location
    Central Arizona
    Posts
    209
    Threads
    12
    Thank You Posts

    Default

    I like the Rosewill enclosure with a 3.5in. HD of your choice. The Rosewill RX-358 V2 3.5" Sata to USB and eSata External Enclosure is the one I'm using.
    Last edited by Alan Melle; 07-02-2011 at 03:18 PM.

  3. #3
    Forum Participant
    Join Date
    Jan 2008
    Location
    Canada
    Posts
    6,588
    Threads
    643
    Thank You Posts

    Default

    I have two Seagate Goflex 3tb drives, one is a master and the other is a Time Machine (Apple) backup. The Sata drive attaches to a base that supports USB2, USB3 or Firewire 800. I have had a lot of very weird things go wrong with this system which has required at least 4 calls to Tech Support. I have not lost anything (amazing) and Seagate have been good, however, I have to say I would NOT recommend this drive. BTW I went with Seagate originally because it had such a good reputation in hard drives. I have wondered if the problems I've experienced are the result of the 3TB size of the drive. Who knows.

  4. #4
    Forum Participant
    Join Date
    Jan 2008
    Location
    Sacramento, CA
    Posts
    3,469
    Threads
    495
    Thank You Posts

    Default

    I am not networking with laptops, just using a desktop. But....

    I would suggest getting a copy of "The Dam Book" on digital asset management. There's so much in the book, much more than you will need but lots of ideas. If you store your captures in chronological order, the "bucket" system seems to be a good way to go. With this system of filing and storage, you fill buckets that are the size of either DVD or BlueRay discs, fill the discs in chrono order, archive or retire them and fill your hard drives and archive them. This archive can build forever (theoritically!). You have to commit to a master file that doesn't change by adding to it or deleting from it for this to work. So, if you keep say, 3 copies of your master file, you would have 2 on hard drives and one on disc. This gives you redundancy and two different storage mediums. I don't know if this is making sense but read the book on it! I don't use this system because my master file needs to be constantly edited as I improve species captures, (but I really like the idea!).
    I am using a 1Tb ingestion C drive, a 2Tb internal (can't remember the models) and 2-1Tb and 1- 2Tb external WD drives, synced with Syncback to the big internal drive. So, I have four copies synced. It is working well for me so far! I like this idea over a raid or drobo because I have redundancy and can pull one of the fully backed up drives and store it off site any time. I would like to eventually burn a copy to Blueray but need to cull down to a really hard core master file first.
    Last edited by Dan Brown; 07-03-2011 at 09:44 AM.

  5. #5
    Forum Participant
    Join Date
    Jan 2008
    Location
    Canada
    Posts
    6,588
    Threads
    643
    Thank You Posts

    Default

    You inadvertently hit the nail on the head Dan- the theoretically sound way to do this may not be the most practical way.

    I view the archival saving of your images as one of the most important unresolved issues in modern photography. One of the many problems is scalability. You may set up a great system for 1TB of images, but it doesn't work a "DAM" if you have 7 or 8 TB. The media themselves are inadequate for the task (DVD and Blueray are not archival so basically forget these; hard drives remain mechanical and thus inherently unreliable devices, and solid state memory is too expensive), so what does one do?

    Your final point Dan is one solution. I think it is safe to say we all save far too many images and we don't ask ourselves often enough what would be your 100, 500, or 1000 "desert island images?". 1000 outstanding images would be a pretty good legacy for a lifetime of nature photography and they would all currently fit on a reasonably sized thumb drive.

  6. #6
    Banned
    Join Date
    Feb 2008
    Location
    Colorado
    Posts
    3,949
    Threads
    254
    Thank You Posts

    Default

    Hi Guys

    I've been out hiking.

    Steve, my first suggestion is why not upgrade your 1 TB drives to 2 TB? That solves your immediate space problem.

    John, I heard reports of some OS's having trouble beyond the 2 TB boundary. I'm surprised Mac is one of them. XP apparently is one (perhaps depending on version). Windows 7 seems OK, although I have not used windows 7 much with 3 TB drives. I have been using dozens of seagate 3 TB drives under linux with zero problems.

    I built the following system partly to solve backup issues:

    I put together 2 I7-950 systems (quad core), one with 12 GBytes ram, the other with 6. I bought a case that can hold 6 drives, and with some adapters, 8. I have 8 USB ports for disks on the motherboard. For my main (the 12-GB ram) desktop, I have a 1.5TB system drive, two 2-terabyte data drives, and two 2-terabyte backup drives. Both systems were under $3K not including the data disks. For the backup system, I have a 1.5 TB system drive and two 2-TB backup drives. Then I have 3 sets of 2-TB USB drives for backup, two of which are kept off site. So on site I have the main system and 3 backups. I keep the backup computer off most of the time to protect it from power surges, especially in summer asd there is a lot of lightning here, and protection against internet attacks. First backup is the the second set of drives in the desktop. Then occasionally I turn on the backup computer and backup to it. I rotate the 3 USB backup sets (more when I have time than a regular backup). I do tend to do a USB backup cycle after a big trip. During photo trips I backup to portable USB drive and that I consider an archive. When the drive is full, I never touch it again--it stays on a shelf, so that is another backup of my photos from my cameras. This USB backup is the dumps from the memory cards on the trips, with few images deleted.

    OK, so now how to manage this. Before my dual system put in place this year, I was running XP. I would do a hand copy of directories of files to a temp USB backup. My 3 backup sets: I would erase one and completely rewrite it for each cycle. A pain and took overnight per disk.

    I'm now in the process of retiring the 2 TB USB drives for 3TB USB3 drives. I have had no problems, and between work and home I have had no problems with any 3 TB drive. I've moved to USB3 for the faster transfer speeds. On XP with USB2, I was getting about 18 megabytes/second. Under linux, I would get 32 megabytes/second (on the same hardware--dual boot). With USB3, and fast disks with linux I get about 96 megabytes/second transfer rates. So I've improved throughput about 5x what I was getting last year.

    Data management is an issue and that added to the reasons for me to move to Linux. For photoshop (since it is not yet on linux) I installed virtualbox and windows 7 (natively) under linux. This gives me the best of both worlds (which for me is Photoshop CS5, ImagesPlus, and PTGui under windows) and everything else under linux.

    I use rsync to backup from my main two drives to my backup drives on my desktop, use rsync to backup to my backup server, and use rsync to backup to my USB drives. It is simple, and very fast, and very very robust. Mac users can get rsyn to run under unix (may already be there). Windows users might try synctoy, which I understand is free from microsoft but I have not tried it.

    An alternative for windows users. When you buy a new PC, convert your old PC to Ubuntu linux and make it a backup server. You can mount the linux disk on your windows machine using Samba services (I did that for many years). You can then easily copy files to the backup server from your windows machine and have better security. You may be surprised how fast linux runs on an old PC.

    Another advantage I have found under linux is the disk health monitoring. The system monitors and reports how each disk is performing, including bad sectors. I have had (between work and home) 2 disks beginning to go bad: increased retries, more bad sectors, but no actual failures. The data presented to me allowed me to make decisions on when to pull a drive out of service before a hard failure. Very impressive. I have pulled 2 (2TB) drives out of about 20 that I use for various projects.

    Note that if your machine is attached to the internet, it is likely getting thousands of attacks per day! A determined hacker might find a way in unless you are diligent about security. Under linux there are tools and firewalls to monitor attacks and block them. And it is all free.

    Roger

  7. #7
    BPN Member Steve Uffman's Avatar
    Join Date
    Jan 2011
    Location
    Baton Rouge, LA
    Posts
    586
    Threads
    77
    Thank You Posts

    Default my solution from the feedback: KISS

    Quote Originally Posted by Roger Clark View Post
    Hi Guys

    I've been out hiking.

    Steve, my first suggestion is why not upgrade your 1 TB drives to 2 TB? That solves your immediate space problem.

    John, I heard reports of some OS's having trouble beyond the 2 TB boundary. I'm surprised Mac is one of them. XP apparently is one (perhaps depending on version). Windows 7 seems OK, although I have not used windows 7 much with 3 TB drives. I have been using dozens of seagate 3 TB drives under linux with zero problems.

    I built the following system partly to solve backup issues:

    I put together 2 I7-950 systems (quad core), one with 12 GBytes ram, the other with 6. I bought a case that can hold 6 drives, and with some adapters, 8. I have 8 USB ports for disks on the motherboard. For my main (the 12-GB ram) desktop, I have a 1.5TB system drive, two 2-terabyte data drives, and two 2-terabyte backup drives. Both systems were under $3K not including the data disks. For the backup system, I have a 1.5 TB system drive and two 2-TB backup drives. Then I have 3 sets of 2-TB USB drives for backup, two of which are kept off site. So on site I have the main system and 3 backups. I keep the backup computer off most of the time to protect it from power surges, especially in summer asd there is a lot of lightning here, and protection against internet attacks. First backup is the the second set of drives in the desktop. Then occasionally I turn on the backup computer and backup to it. I rotate the 3 USB backup sets (more when I have time than a regular backup). I do tend to do a USB backup cycle after a big trip. During photo trips I backup to portable USB drive and that I consider an archive. When the drive is full, I never touch it again--it stays on a shelf, so that is another backup of my photos from my cameras. This USB backup is the dumps from the memory cards on the trips, with few images deleted.

    OK, so now how to manage this. Before my dual system put in place this year, I was running XP. I would do a hand copy of directories of files to a temp USB backup. My 3 backup sets: I would erase one and completely rewrite it for each cycle. A pain and took overnight per disk.

    I'm now in the process of retiring the 2 TB USB drives for 3TB USB3 drives. I have had no problems, and between work and home I have had no problems with any 3 TB drive. I've moved to USB3 for the faster transfer speeds. On XP with USB2, I was getting about 18 megabytes/second. Under linux, I would get 32 megabytes/second (on the same hardware--dual boot). With USB3, and fast disks with linux I get about 96 megabytes/second transfer rates. So I've improved throughput about 5x what I was getting last year.

    Data management is an issue and that added to the reasons for me to move to Linux. For photoshop (since it is not yet on linux) I installed virtualbox and windows 7 (natively) under linux. This gives me the best of both worlds (which for me is Photoshop CS5, ImagesPlus, and PTGui under windows) and everything else under linux.

    I use rsync to backup from my main two drives to my backup drives on my desktop, use rsync to backup to my backup server, and use rsync to backup to my USB drives. It is simple, and very fast, and very very robust. Mac users can get rsyn to run under unix (may already be there). Windows users might try synctoy, which I understand is free from microsoft but I have not tried it.

    An alternative for windows users. When you buy a new PC, convert your old PC to Ubuntu linux and make it a backup server. You can mount the linux disk on your windows machine using Samba services (I did that for many years). You can then easily copy files to the backup server from your windows machine and have better security. You may be surprised how fast linux runs on an old PC.

    Another advantage I have found under linux is the disk health monitoring. The system monitors and reports how each disk is performing, including bad sectors. I have had (between work and home) 2 disks beginning to go bad: increased retries, more bad sectors, but no actual failures. The data presented to me allowed me to make decisions on when to pull a drive out of service before a hard failure. Very impressive. I have pulled 2 (2TB) drives out of about 20 that I use for various projects.

    Note that if your machine is attached to the internet, it is likely getting thousands of attacks per day! A determined hacker might find a way in unless you are diligent about security. Under linux there are tools and firewalls to monitor attacks and block them. And it is all free.

    Roger

    thanks to everyone for their help....here is what I did

    installed a new 2tb wdc caviar black 7200rpm drive in the desktop and then moved all images to it....

    installed a dual port usb 3.0 card in the desktop

    took the wdc my book 2tb out as a backup and replaced it with a wdc my book 3tb drive attached to the usb 3.0 port (now have a total of 3tb on the desktop and 3tb of backup)

    repurposed the 2tb wdc my book as the "other" drive that I have LR3 also write to when I import images

    also bought a wdc 1tb to back up my Lenovo x220 notebook which is a terrific machine. It also has a usb3.0 port so my catalog exports from it to the desktop are very quick

    added note:on my laptop I use a delkin 3.0usb card reader. works fine but which it had a cover like the Lexmar....have had lint get into when traveling

    I do not have the offsite backup solution other than I transmit my best images to Smugmug. am going to look at their smugvault....while as a hobbyist, I don't depend on my images for income, my stakeholder risk is high since my wife would kill me if I lost a single image of our kids or grandchildren

  8. #8
    Co-Founder James Shadle's Avatar
    Join Date
    Oct 2007
    Location
    Valrico, Fl
    Posts
    5,108
    Threads
    1,419
    Thank You Posts
    Blog Entries
    11

    Default

    USB 3.0 - that's all you need to know.
    What a difference!!!!

  9. #9
    BPN Member Steve Uffman's Avatar
    Join Date
    Jan 2011
    Location
    Baton Rouge, LA
    Posts
    586
    Threads
    77
    Thank You Posts

    Default usb 3.0 is a HUGE improvement and an upgrade that might be simple and cheap (it was for me)

    Quote Originally Posted by James Shadle View Post
    USB 3.0 - that's all you need to know.
    What a difference!!!!
    underestimated how big an impact until I got my new notebook with USB 3.0 and a faster drive Laptop rocks but even so USB 3.0 seems like a game changer....It absolutely rocks....and people may not realize that the cards to upgrade might not be too expensive...for my desktop, it was $29 for a dual port. Of course, I checked with Dell support to be sure my machine could handle it and they identified the part I needed....

  10. #10
    BPN Viewer
    Join Date
    Jan 2009
    Location
    Rio de Janeiro - BRazil
    Posts
    202
    Threads
    42
    Thank You Posts

    Default

    I don't know how "techie" you are but maybe you should consider a NAS.

    https://secure.wikimedia.org/wikiped...tached_storage

    I've been researching about it and some brands that I would consider are:
    Synology, NETGEAR, Thecus and QNAP.

    I hope this helps.

  11. #11
    BPN Member Steve Uffman's Avatar
    Join Date
    Jan 2011
    Location
    Baton Rouge, LA
    Posts
    586
    Threads
    77
    Thank You Posts

    Default LR3 and NAS seemed problematic

    Thanks, I looked heavily at NAS and would have gone that way BUT- I use LR3 and there were too many questions about whether it would work properly under that environment. In the end, we went with a simpler solution.

    We were going with Netgear if we went that way. And likely will go that way in the future as the technology evolves and Adobe builds LR3 in a way that will work in a straightforward way in that environment.

  12. #12
    BPN Member Steve Uffman's Avatar
    Join Date
    Jan 2011
    Location
    Baton Rouge, LA
    Posts
    586
    Threads
    77
    Thank You Posts

    Default WD 3T MY BOOK Essential USB 3.O might not be my answer

    First bear in mind that I started a technology company in 1978 that I sold to a Fortune 500 company a few years back..grew up with the industry and we ran a very large data center as an ASP that had no downtime during Katrina even though we were located in South Central Louisiana. We served an international industry that could not afford any downtime regardless of circumstance

    Take that experience and couple it with this HD back up fiasco I have been going through and I feel like I am back in the early 80s with a relative immature industry. A 3tb hard drive that works is really not rocket science

    Here's my point-Even though the box for the WD My book Essential 3TB USB 3.0 + 2.0 says on the outside that it is compatible and comes formatted NTFS for Windows XP, Vista and 7, my experience is that I got frequent I/O errors and have not had a successful backup with it.

    Researching I found two interesting posts on forums you should read before you consider this option
    1)http://answers.microsoft.com/en-us/w...c-68b599b31bf5
    2 )http://wdc.custhelp.com/app/answers/detail/a_id/3868/

    posting this scenario is only to hopefully save someone some headaches and time down the line

  13. #13
    BPN Member Steve Uffman's Avatar
    Join Date
    Jan 2011
    Location
    Baton Rouge, LA
    Posts
    586
    Threads
    77
    Thank You Posts

    Default

    Quote Originally Posted by Dan Brown View Post
    I am not networking with laptops, just using a desktop. But....

    I would suggest getting a copy of "The Dam Book" on digital asset management. There's so much in the book, much more than you will need but lots of ideas. If you store your captures in chronological order, the "bucket" system seems to be a good way to go. With this system of filing and storage, you fill buckets that are the size of either DVD or BlueRay discs, fill the discs in chrono order, archive or retire them and fill your hard drives and archive them. This archive can build forever (theoritically!). You have to commit to a master file that doesn't change by adding to it or deleting from it for this to work. So, if you keep say, 3 copies of your master file, you would have 2 on hard drives and one on disc. This gives you redundancy and two different storage mediums. I don't know if this is making sense but read the book on it! I don't use this system because my master file needs to be constantly edited as I improve species captures, (but I really like the idea!).
    I am using a 1Tb ingestion C drive, a 2Tb internal (can't remember the models) and 2-1Tb and 1- 2Tb external WD drives, synced with Syncback to the big internal drive. So, I have four copies synced. It is working well for me so far! I like this idea over a raid or drobo because I have redundancy and can pull one of the fully backed up drives and store it off site any time. I would like to eventually burn a copy to Blueray but need to cull down to a really hard core master file first.

    My system will be if I ever get the WDC 3tbs to work properly with windows 7 somewhat like yours. I have a 1TB internal for OS, and now a 2tb WDC caviar black with all my images. My plan is to back up to two 3tb drives that I will periodically rotate with one being offsite at all times. I also have a 2 TB USB 2.0 that I simultaneously write my images to when I import into Lightroom.

    Hopefully, after 5 days of working on this thing, it will be fully in place today. Would have been simple if the 3tbs from WDC disclosed that they were not out of the box plugin as advertised and I would have known the simple way to move LR3 to a second drive

  14. #14
    BPN Member Steve Uffman's Avatar
    Join Date
    Jan 2011
    Location
    Baton Rouge, LA
    Posts
    586
    Threads
    77
    Thank You Posts

    Default WDC 3tb External Service Bulletin Critical to getting Drive Up

    WDC 3tb USB 3.0 and drives 2.5 TB + come formatted for Windows 7 in a matter that will not work. The WDquick formatter included to format for Windows does not work....However, you can download a correct formatter from WDC as mentioned in the service bulletin that makes it purr like a kitten. Hopes this saves someone some time down the road.

    All other WDC drives I have used have been simple plugnplay. I am sure this one will in time once they get their QA on the installation process done properly

  15. #15
    BPN Member Steve Uffman's Avatar
    Join Date
    Jan 2011
    Location
    Baton Rouge, LA
    Posts
    586
    Threads
    77
    Thank You Posts

    Default beware of WD smartware and windows 7 backup conflicts. Killed WDFME.exe process and life got better

    One hopefully final log on getting the new backup/laptop plans in place....hopefully this will someday save someone a headache

    WDC drives that I have all come with WDsmartware which is their backup utility.....probably a good program in its own right but i chose to use the window 7 backup/restore programs.....which had worked like a charm until I added a bit more complexity (albeit not much)
    My plan was to have two 3tb external drives to back up my 1tb + 2tb desktop....I would rotate one of the 3tbs offsite....I also had a 2tb external that I make extra copies of images, documents and etc....(belt and suspenders, if you will)
    I also use a 1tb WDC passport to save/backup images in the field from my cameras and notebook...and it also serves as a fine transfer tool....

    All was working well until somehow the WDsmartware software became active on my desktop...not sure if it was a leftover from the formatting issues with the WDC external drives 2.5 TB+ or if I populated it on my desktop from the 1tb passport....anyway, once it became active, my backups starting failing (often after 10-12 hours of running) with a message specified file not found....with all the changes we had made we went through eliminating the hardware changes, use a restore point for a system image and more...still no success....

    Then I looked at my resource monitor, and the WD smartware File Managment Engine was consuming all the available memory on my desktop (8gb) and my hard faults were 100%..no wonder things were that slow....and it is likely that WD smartware and windows 7 ran into record conflicts...anyway discovery is 90% of the problem...so went into Services.msc and did the following



    1) Launch services.msc (Start menu -> Search -> services.msc)

    2) Stop WD File Management Engine (Right click -> Stop)
    3) Change service from Automatic to Manual (Right click -> Properties -> Startup type: Manual)
    4) From now on when you want it to startup, simply start service via services.msc. You can stop it via Task Manager or service.msc.

    If you have trouble, you might research it but this fixed my problem


  16. #16
    Banned
    Join Date
    Feb 2008
    Location
    Colorado
    Posts
    3,949
    Threads
    254
    Thank You Posts

    Default

    Steve,

    Thanks for the info. Did you get a good backup?

    Roger

  17. #17
    BPN Member Steve Uffman's Avatar
    Join Date
    Jan 2011
    Location
    Baton Rouge, LA
    Posts
    586
    Threads
    77
    Thank You Posts

    Default

    Yes I did and in a much shorter time...I know these posts apply to only a narrow group but if it helps anyone, just my small way of paying back for all the good stuff I get off of here...also, my wife's Lenovo y570 has 4 USB ports...2 of which are 3.0

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Web Analytics