Compact Flash Card speed problems

[Roger Clark]

Hi,
I'm really confused!

I bought a new Sandisk Extreme IV UDMA card and tested it in my Canon 1D Mark II.
The results are at:
http://www.clarkvision.com/imagedeta...ash_card_speed

Out of the box I ran tests monitoring the write speed as the card filled up. Great results, and pretty much the same write speed right up until the card was full. (Blue curve on figure 1 on the web page).

The I reformatted it in camera and did the test with raw + jpeg. Poor results. (Green curve on the web page).

Finally, I reformatted again in camera and did the test again with raw only. The results should have been equal to the first, but are pretty poor. (magenta curve on the web page).

It seems that the Canon card format might be different than the original Sandisk format. After the camera formatted the card, the format said FAT32. Does anyone know what the format on the card is out of the box?

I have noticed a slow-down of write speed as cards approach full with all my Canon cameras. Have you noticed this issue with 1D III models?

Any ideas what is going on? I want the performance I got out of the box. Any idea how to get it back?


Roger

[Roger Clark]

I have contacted sandisk tech support. They told me the original format out of the box is FAT32 then started asking questions about my computer. I had to reiterate that the problem I illustrated has nothing to do with any computer, except the one in the camera. At their suggestion, I reformatted the card on my computer (FAT32) and got the same speed results as the in-camera formatted FAT32.

So could the be a case of the block size used in the formatting? Or maybe flash cards write fast the first time used, then slow down after that? Ideas?

Roger

[Grady Weed]

Roger,

Try this; buy another card of same size, manufacturer etc, then use it to take a series of images. Then just delete them, do not format the card. Then take some more and delete them. Then compare write speeds to the original purchased card now formatted in fat32. If you just delete rather than format, then you can tell if the issues are camera or card and definitely not PC.

[Grady Weed]

Roger,

I use a Canon 5D, Canon MK2n, 8 gig 133x Lexar Pro cards. I rarely fill up the cards however. Most times I take my images, then download them to my custom built PC, that's what I do, then format the card in camera, after checking my images for ones to keep. I rarely just delete them, only mostly when I check them in camera for exposure and sharpness, then dump the ones I do not want. I have not noticed my card writing slower due to my formatting rather than deleting the images. I just might go out and buy a card to try it however. Maybe you have something the card makers want to try and cover over? Sounds interesting.

[Roger Clark]

Hello,
I have heard from Sandisk support. They suggest I return the card and get a new one. I'll do that, and I'll test it multiple times without reformatting it to see if I get any slowdown. I'm skeptical that it is the card. The last time I contacted sandisk support was over an MP3 player that didn't work. They strung me on for months of do this, do that denying there was a problem. Then they came out with a firmware fix that solved the problem.

Has anyone else noticed a slow-down of their camera's writing images as the card gets full?
Roger

[Dave Phillips]

when card gets fuller, it may be seeking open fragmented space to write to.....and taking time to do so.

Memory cards do fragment.....right?

[Roger Clark]

when card gets fuller, it may be seeking open fragmented space to write to.....and taking time to do so.

Memory cards do fragment.....right?

Dave, yes in theory if you erase and write files. That is why I reformat. With a new file system, there are no fragments. Also, formatting is faster than deleting files to clear space on a full card.

Roger

[arash_hazeghi]

This is very odd, the format for all FLASH cards on the market is FAT32. Flash memory is random access with row and column adressing exactly like RAM so it should not slow down as you fill it up. what is your test method? how are you measuring read and write speed?

[Chris Starbuck]

Roger,
Speaking as an electronics engineer who's been using flash memory for as long as the technology has been in existence, I think what you're seeing is some performance limitation of the 1D2 that has almost nothing to do with the CF cards. Flash memory does eventually slow down with use, but only after 10,000 to 100,000 reformats, and even then the slowdown is a few percent per 1000 reformats. The fact that you see similar results with different brands and different speed grades of CF cards supports this. The fact that jpegs seem to be slower to write than raw is because the raw files require very little processing between the A/D converter and writing the file (just creating the very small embedded jpeg), other than aggregating the data into blocks, while creating a full-resolution jpeg from the raw data requires a great deal of processing. The lack of a speed benefit from the new UDMA card over the older 133x card is also a factor of the camera design rather than the CF card; the old 1D2 simply can't supply data as fast as the new UDMA cards can write. As Arash pointed out, flash memory access is random, just like RAM, and access time does not depend on location, nor how full the card is. Even if the space on the card were fragmented (which is not the case with a reformat, as you correctly point out), it isn't the CF card that's doing the file management - it's the embedded computer in the camera.

A possible insight (forgive me if this is stuff you already know; I'm assuming it's unfamiliar to at least some of the readers): In a FAT (file allocation table) file system a chunk of space is reserved at the physical beginning of the medium (disk, flash memory, etc) for the FAT. This table contains one entry for each physical storage block on the device (excluding the space reserved for the FAT itself), in sequential order. Each entry in the FAT contains either a pointer to the next used block (FAT entry) in a file, or a tag indicating the block is unused. Starting from a blank, reformatted card, all blocks are marked "unused" in the FAT. Each time the camera CPU needs a new block of space to write to, it has to search the FAT for the next available block. The most reliable (and simplest/dumbest) way to do this is to search linearly from the beginning of the FAT until it finds a block marked "unused". As the card fills up, more of the FAT has to be read in order to find the next available block. So the time spent reading the FAT increases linearly as the memory fills up. Offhand I can think of at least one software technique that could be used to speed this up a great deal, but I have no idea whether Canon has implemented such a thing. Still, the FAT is a tiny portion of the whole device, and I can't believe the increasing FAT read time could account for the dramatic slowdown you've seen. It would be interesting to run the same tests, with the same CF cards, in one of the latest camera models.

You might be able to get more insight into what's happening by talking to Canon, though I rather doubt their tech support folks have access to this sort of detailed design information.

[Roger Clark]

Arash, Chris,

Multiple things here. Testing speed is quite simple. Here is what I do. Start with an empty card. Put the camera in multi-shot mode, IS off, AF off. With the camera pointed at a bright target giving a fast shutter time (faster than 1/500 sec.) and with one hand on the shutter and the other on a stopwatch, press both butting simultaneously. Hold the shutter down and fire off many frames, You can go to buffer full, or stop at 10 to 20 frames. How many you do does not affect results except the more frames you take the longer the time to write them and the less the error is in operating the stopwatch. Watch the writing to card light on the camera then stop the stopwatch at that moment. After doing this a few times the timing error is less than about 0.1 to 0.2 second.

Next connect the card to a computer and record how many files you wrote, and their total size. Compute the result:
write speed = total size of the files written / time from stopwatch.

The numbers I get at
http://www.clarkvision.com/imagedeta...ash_card_speed

are the same as what Rob Galbraith gets:
http://www.robgalbraith.com/bins/mul...e.asp?cid=6007
when the card is empty. I have written Rob (well the contact info on the web site) but have received no response.

Next fill the card with more images. You can do this with your normal shooting and periodically do the test, or copy images from the computer, e.g. sets of images you've previously taken with the same camera. I'vce tested both ways and it makes no difference in the results.

Back to the problem. The write speed slow down is not just a problem with the 1D2. I've confirmed it on Canon 10D and 30D cameras. One would think the faster cards would be more limited than older cameras like the 1D2 so one would think the problem would be less. The Extreme IV UDMA card does work faster than 133x cards in the 1D2, so it is not simply a camera limitation. The empty card speed on the 1D2 for a lexar 133x card is about 6.8 MBytes/sec whereas the Extreme IV UDMA gives 7.5 MBytes/sec. I had hoped the faster card would do better as the card fills. But as a percentage of full, it does worse.

I understand file systems (I have administered multiple machine for 25+ years, mostly unix and linux). The only things I can think of is the time to find the unallocated blocks increases as the card fills. But as I understand FAT, as each file is written, the directory entry must be written too. So there are multiple access that must occur, reading from one location (to find unallocated blocks), and writing to two locations (the file and the directory entry), and as the card fills, these are further apart.

What I don't understand is why out of the box the speed was so good then after reformatting, performance dropped so much. Last night I ordered 3 more Extreme IV UDMA cards (great rebate going on now $150 on 3 cards). So I'll test them without reformatting them. If that makes a difference (perhaps the block size is different?), I'll no longer reformat a card. But reformatting is a faster way to clear a card for use, and provides a new file system that is less likely to develop problems.

So for the curious engineer types out there, please do some speed tests and see if you can confirm the slow-down effect. Knowing about the problem allows you to at least mitigate the effects if you are in a rapid shooting situation.

Roger

[Chris Starbuck]

Roger,
Galbraith's site is interesting; hadn't seen it before (but hadn't had any occasion to look for this kind of data). Anyway, for the Canon 50D with the Lexar Professional 300X 4GB (UDMA) he reports a RAW write rate of 29.985MB/sec and a JPEG write rate of 18.433MB/sec; his test only writes 20 jpegs, then 15 RAW files to a freshly formatted card).

I don't have a stopwatch, so had to just watch the second hand on a clock, press the shutter release (in high-speed continuous mode) as the second hand hit a reference mark, stop 1 or 2 shots after buffer full (noted by the sudden slow-down in shooting rate), then note the elapsed time when the write LED went off; rounded up to the next whole second. I was recording RAW + large fine JPEG. Shutter speed was faster than 1/1000. Each such burst was 11-13 exposures (22-26 files, 280-361 MB) and took 8-13 seconds. With uncertainty in start/stop times I could have about a 10% measurement error, but that's much smaller than the changes in write speed that you're getting. Anyway, for each such burst I recorded the beginning and ending image # and elapsed time. Starting from a freshly formatted (in camera) CF card, I sampled 3-4 bursts at the beginning, 1 burst when the card was about 1/2 full, then 2-4 bursts when the card was nearly full; quit when the space remaining was less than 10 shots. Repeated this (filled up the card) 3 times. After each fill-up, stuck the card in a reader and calculated the total file size for each burst, using the previously recorded image #s. Put my data in a spreadsheet, calculated write rate for each burst, and calculated mean & standard deviation for each fill-up, and overall for the 3 sets. The write rates varied from one burst to another, but not monotonically and not showing any trend from empty to full card, nor any difference among the 3 sets. For 22 samples (bursts) my mean write rate was 31.63 MB/sec, with a standard deviation of 2.18. All samples were within 1.8 standard deviations of the mean (all but 4 were within 1.5). Note with the 50D raw files range from about 20-30 MB and the large fine JPEGs are about 6-10 MB. Definitely not seeing any slowdown with my setup.

Given that you've seen the same slowdown behavior on 3 different camera models, as well as 2 different types/brands of CF card, I'm out of ideas.