content aware scaling instead of cropping for stitched panoramas

[Steve Gould]

Hi,
I'm a brand new memer of BPN, although I have read numerous threads over the past months, and this is my first post. I am working on some stitched panoramas in Photoshop CS4, and the stitching always gives non-even edges. Normally I just crop to a rectangle. But with these, if I crop to a rectangle I will lose some important elements near the edges. I believe sometime ago I saw a way to use Content Aware Scaling in CS4 to pull down and even the edges in order to save all the elements, but I can't find it at Adobe or on the web. Does anyone know how to do this?
I hope I'm doing the posting correctly; please correct the newbie, if necessary.
Thanks!

[Roger Clark]

Steve,
Welcome!

I don't have CS4 with me at the moment, but I think you may be talking about text and other graphics objects when an image is resized. I have seen some videos on image object content aware scaling that may be in CS5.

Regarding mosaicking, I assume you are referring to the outer edges, which often become curved. If so if you want to keep something in the corners, as well as something in the middle at the edge where the corners stick out further than the middle, then the only solution I know is to select a portion of the image, feather it (with the selection on the edge side of the image well beyond the image--you may need to increase canvas size to do this) then use the edit -> transform -> skew or distort (in photoshop; not sure about elements). This can work OK for small distortions, but can cause artifacts if pushed too far. You could do different selections and small distortions so not one step distorts the image too much.

A better solution is to take extra images when making the mosaic. For example, if I want a 2x3 frame mosaic, I'll do those plus 4 more images: at top and bottom and left and right to fill in the curved edge that would show after the images are warped. The curved edges are a natural product of projecting the curved sphere that you image onto a flat plane.

Or maybe I'm misunderstanding your question. If so, how about posting an image showing us the problem.

Roger

[Steve Gould]

Roger,
Thanks for responding quickly. I'm attaching the image. It was created from 5 vertical captures stitched together with CS4. I've done no processing in Lightroom or in Photoshop other than the stitch - or mosaicking. I made sure I had room on both sides to capture the set of fur seals for the foreground. And, I clearly had plenty of room at the top to have shifted the camera in order to get more room at the bottom, but I suspect I was too awed by the scene to set things up properly. So, now I'm left with trying to salvage it. What do you think?
Thanks

[Mike Tracy]

I don't have cs4 and not sure if this is the direction you wanted to head. Three minute patch tool and clone stamp with a minor recrop to get a couple of birds of the edges. If you do nothing else get rid of the guy in the red on the right center (I did). I could have done it perfectly but tired and heading to bed.

[Jay Gould]

Mike, great RP. Agree with the crop on both sides as you have opened up that first line of penguins. I would consider removing some of the sky since there is nothing there but negative space and consider more of a panorama. I did a scrolling crop and removed 50% of the sky between the top and the highest peak.

Looking forward to seeing the image posted after PS processing. TFS! :D

[Roger Clark]

Hi Steve,
I just got home from Houston--long trip with flight delays.

That is a great image (wish I was there). Mikes clone solution is very good and a reasonable solution. Here is what I would do. I would select between the penguins and the 4 sea lions on the lower left and the selection pie-shaped to the bottom center (going around the animals). Then use the distort tool to pull down the image to the bottom of the canvas. Any small left-over white space could be filled in by cloning. I would also select, feather and darken the sky and mountains with the curves tool.

Roger

[Steve Gould]

Hi Mike,

Thanks for your response, and Jay - thanks for your comments. The situation isn't so simple, and Mike's repair doesn't work for me. I print and sell my images, so everything has to be flawless. The native size of this image at the moment is about 13x30 at 360 ppi (what I print at with my Epson 7800), so any duplications or flaws in the cobbles will be very noticeable. That's very different from a 1024 pixel image on a computer monitor. Mike, you patched on the right side using image from the middle, and that doesn't match the much larger cobbles on the right in the original. I spent over 2 hours last night doing my first repair effort using 3 sections via Robert O'Toole's Quick Mask technique, followed by loads of healing brush and stamp clone tool to get rid of all the duplications. With an area with as much detail as I have in the foreground, the duplications are very obvious and have to be repaired. That kind of time was what I was hoping to avoid. As far as removing the guy in the red parka and crop issues with the sides, and actually processing the image with curves such as Roger mentions, etc., these are all relatively straightforward things for me and I wouldn't do them unless I knew I had an acceptable solution to keep all the fur seals at the bottom. Anyway, I'm attaching my first repair effort from last night, and I did remove the guy in the red parka, as well as a small orange parka near him, and I removed the partial penguin on the left instead of cropping that. I have a guide that is just above the remaining white at the bottom that is where I will do the final crop. However, I just noticed that there is still at least a pair of cobbles near the bottom below the middle seal pup on the right that have to be fixed.

I did pose my original question to another photography group that I belong to, and I got two responses. One was not useful and the other is reproduced here; I will try it to see if it does the job better:

"Duplicate a crop area of the edge you want to extend. Use content aware scale just on the cropped area, then use a layer mask on the scaled crop to protect any the subjects that may have still been slightly distorted by the CAS. The results really all depends on how complex the image is."

Roger, thanks for your suggestion, and I'll try that, too. I want the very best look I can get, and all the healing and stamping I had to do may not be the best result.

Thanks, all, again.

[Michael Gerald-Yamasaki]

Steve,

Greetings. A couple other approaches you might try:

- Selectively warp the image (it's warped a bit anyway from the photomerge and would be with any auto-scaling tool).

- Select a crop to transform putting the blending seam in a place where cloning/merging is less obvious.

- Try a different stitcher (such a ptGUI) which offers more options for stitching (greater selection of transforms during stitch). If you're goal is for sales, you probably want a better stitcher anyway (JMO).

Cheers,

-Michael-

[Lance Peters]

IMHO - like everything - get as much right to start with and less to do after the event. (Not going to help with this one but will in the future)
Are you using a Panoramic head on your tripod?? Ive used the Nodal Ninja's for a while now - not too expensive and do a good job.
Second if you are using a pano head - do you have your settings correct?? Has to rotate around the Nodal point of the lens (Different for different lenses) otherwise you will get Parallax errors - also has to be level!!

PTGUI is the best stitcher I have found - does a reasonable job on auto and can be tweaked manually - I use this for virtual tours for real estate agents.

[Jay Gould]

Lance, I assume you are using the Pro version?! I think it is time to breakdown and add to my software and equipment.

[Roger Clark]

Jay,
I'm using ptgui pro. I tried photoshop's stitcher but didn't like it nor the artifacts. PTGui gives you much more control, especially for mosaicking moving subjects.

Roger

[Steve Gould]

Lance,
I'm using a RRS ballhead. I usually don't shoot panoramas and I'm usually hiking for much of my photography, so I do limit the amount of equipment I'm carrying - was the same for this trip to Falklands, S. Georgia Is. (where this was shot), and Antarctica. I will look into PTGui Pro; I have heard of it before.
Clearly, I should have taken more images for the foreground, and that's a lesson I hope I remember!
Thanks.

[Lance Peters]

Lance, I assume you are using the Pro version?! I think it is time to breakdown and add to my software and equipment.

yES jAY!!

[Lance Peters]

Lance,
I'm using a RRS ballhead. I usually don't shoot panoramas and I'm usually hiking for much of my photography, so I do limit the amount of equipment I'm carrying - was the same for this trip to Falklands, S. Georgia Is. (where this was shot), and Antarctica. I will look into PTGui Pro; I have heard of it before.
Clearly, I should have taken more images for the foreground, and that's a lesson I hope I remember!
Thanks.

Take a look at the nodal Ninja pano head - not all that heavy - very light in fact and does a good job.
WWW.NODALNINJA.COM - Nothing to do with me in case you were wondering.

[Steve Gould]

Michael,
Would you point out where you think you see the seams in the CS4 stitch? After seeing your post, I went back to the stacked layers and examined each seam. I don't believe you can see the seams. You may be noticing some lines of cobbles, but they are not at the seams. This cobble beach is exposed to some pretty wild weather. We had to evacuate after only 4 hours there, instead of getting the whole day, because a storm had come in and we were in danger of not being able to get the zodiacs to/from the beach and back to the ship. So, the surf at high tide can really move those cobbles around and the lines are the results of that.

Thanks to all the additional c&c's. I've done some extensive tests are here are the results. Please don't comment again on the red parka, etc. I'm only concerned at this point with getting the best bottom and dealing with the rest of the image processing will be straightforward for me.
I tried PTGui Pro in the trial version, so the image has the watermark all over it, and I pretty much just let it do its thing. I then looked at various projections. PTGui did give a slightly better bottom than had CS4 cylindrical, and I settled on PTGui rectangular. However, as you can see in the image, it did a poor job when aligning/blending images 4 and 5 in the stitch, and left me with a ghost of part of the middle seal pup. CS4 did not have that problem and gave a clean alignment/blend. Fortunately, it is suficiently isolated that I could take a patch from the same area of the CS4 stitch and repair it. So, PTGui is not the answer for everything, and I believe we need a well-stocked tool kit. FOR SOME REASON, BPN IS NOT SHOWING THE IMAGE EVEN THOUGH IT SEEMS TO ACCEPT THE UPLOAD. I WILL POST THE IMAGE IN A MOMENT IN A SECOND POST.

[Steve Gould]

My error, I did the Browse correctly but forgot to click on "Upload". Okay, here is the portion of the image from the PTGui stitch that shows the ghost of the seal pup.

[Steve Gould]

Now I will show you the full PTGui panorama WITH the use of Content Aware Scaling to fix the bottom right side. Here's how I did it; it is complicated but it gave a very nice result.
I did figure out how to use the content aware scaling, so what you see in the attached image has no duplicated cobbles, no need to heal and clone stamp. In the following, it's important to save and name all the selections so you can call them back when you need to. The trick was to duplicate the background layer, select only about 70% of the image - the left side was just where the lack of image pixels at the bottom began. Then invoking CAS on that, protecting all the seals with the mask I had first created, and then pulling the selection down to the crop line I had picked before. That worked beautifully for that 70% of the image, but now there was a radical vertical discontinuity on the left edge with the remaining original 30%. So, I now duplicated the background layer again, called back the 70% selection and inverted it, then removed the lower portion that had the seals, and then did a control-J to move that to a separate layer. Now, I just moved this piece down until everything lined up. There was a little clean up needed along the lower edge using a layer mask. Finally, I did crop the edges, including removing some of the excess sky.
While this process gave me a result I like, it did take quite a while to do. Yes, I did it on the trial version so the watermark is all over it. But, since I saved all my selections I can redo everything quickly when I buy the Pro version.
However, in the next post I will show you what I got with Transform>Warp with just about 3 minutes' effort.

[Steve Gould]

Continuing my learning with this panorama, with the hope that you will also appreciate and learn from the different results, here is the repaired and cropped image obtained in about 3 minutes, beginning with teh CS4 stitch using the cylindrical projection. It was created simply by taking the flattened file and going to Edit>Transform>Warp, and then pulling down on the right and pushing up on the middle and a little pushing up on the right, followed by a little pulling on the sides to straighten things out.
Only 3 minutes, and no need to patch, duplication of anything, no need for layer masks. The tiny bit of white canvas on the bottom edge will be trivial to repair, but I wanted to show the result of the Warp.

[Steve Gould]

Here's the last one. Again, I began with the stitch from CS4 cylindrical. I followed Roger's suggestion and first did a selection of the bottom portion of the image, exclusive of all the animals except the group of seals on the right (and saved the selection). I then went to Edit>Transform>Distort, and found that when I pulled the selection down I got empty canvas where the pixels moved from. So, I started again and duplicated the background layer, and then did the Distort on the duplicate. That worked pretty well, except it portion that was moved by Distort covered up some parts of the other animals. So, I had to repair that with a layer mask to paint with black to bring those parts back. In addition to this extra work compared to using Warp, the seals on the right are slightly - but noticeably - larger than they were before the Distort, and they are shifted in their horizontal position relative to the seals on the left. AND, with all the repairs it took over 30 minutes to get a result I'm happy with.
I would be interested in hearing which version you like best: the first (lots of patching followed by lots of healing and stamping, but it did give more image below the seal pups), the second one done with CAS (but took over an hour to get it right), the third one done with Warp (3 minutes), or the last one done with Distort. Admittedly, these were my first efforts with CAS, Warp, and Distort, so not only might I learn to do use these faster but some of you might be able to use them a lot better than I did.

[Roger Clark]

Steve,
I like the one in panel 17.

Some notes. The ghost image of the seal appears to be because the seal moved between the two frames. There is an additional program you can get with PTGui called smart blend. It tries to locate conditions of moving subjects and compensates. Also, you can output the mosaic as photoshop layers so you can change the seams between images. I usually have ptgui output both the blended mosaic and the photoshop layers in case I need to fix something.

Regarding tripod heads, this is a classic landscape mosaic where the camera should be horizontal and not vertical as most pano heads seem to require. The reason for horizontal is the vertical dimension is smaller so you control depth of field better. My simple but effective light pano head is a 6-inch Wimberly plate with a Wimberly clamp on one end. I can put the plate on the clamp on my ball head then the camera one the second clamp and slide the camera+lens back and forth to the nodal point. The nodal point of course only works in the azimuth scan (left right movement). So in this pano I would do a quick pano down low. With animals moving, I look through the viewfinder and move the camera as needed to frame around moving subjects, then I would do the second row above the first, letting the focus move to the far horizon. Because the row above the first is further away, the parallax induced by the ball head upward rotation is insignificant. I have never had an issue with parallax.

Roger

[Jay Gould]

Steve,
I like the one in panel 17.

Some notes. The ghost image of the seal appears to be because the seal moved between the two frames. There is an additional program you can get with PTGui called smart blend. It tries to locate conditions of moving subjects and compensates. Also, you can output the mosaic as photoshop layers so you can change the seams between images. I usually have ptgui output both the blended mosaic and the photoshop layers in case I need to fix something.

Regarding tripod heads, this is a classic landscape mosaic where the camera should be horizontal and not vertical as most pano heads seem to require. The reason for horizontal is the vertical dimension is smaller so you control depth of field better. My simple but effective light pano head is a 6-inch Wimberly plate with a Wimberly clamp on one end. I can put the plate on the clamp on my ball head then the camera one the second clamp and slide the camera+lens back and forth to the nodal point. The nodal point of course only works in the azimuth scan (left right movement). So in this pano I would do a quick pano down low. With animals moving, I look through the viewfinder and move the camera as needed to frame around moving subjects, then I would do the second row above the first, letting the focus move to the far horizon. Because the row above the first is further away, the parallax induced by the ball head upward rotation is insignificant. I have never had an issue with parallax.

Roger

Hi Mate, why do you prefer 17 over 18?

I just purchased PTGui Pro; where do you find the smart blend add-on?

Regarding your home made pano head, could you post some images? It sounds like something similar in use to the Sidekick (which I am so very glad you convinced me to purchase!) in that you do not remove the ball head.

:D

[Roger Clark]

Hi Mate, why do you prefer 17 over 18?

I just purchased PTGui Pro; where do you find the smart blend add-on?

Regarding your home made pano head, could you post some images? It sounds like something similar in use to the Sidekick (which I am so very glad you convinced me to purchase!) in that you do not remove the ball head.

:D

Jay,

I like 17 because the crop on the sky is lower. I feel the large white sky in the others is a distraction. Second, the crop on the left edge is not cutting off any birds, but it does in the others. I would have added one more frame on the right so that the curving arc of penguins was not broken.

I got the smart blend from the ptgui web site, or at least there was a link from the ptgui web site. It does add significant compute time to the processing.

I'll see about posting some images in the gear forum. It might be a couple of days--I have lots of stuff due this week. Please remind me if you don't see it by later this week.

Roger

[Steve Gould]

Roger,

Thanks for you continuing responses. The crop on the left is not critical at this point. #17 is clean there relative to the others only because I haven't done the final crop other than at the bottom, but thanks for mentioning it in case I forgot to take care of that. I will clone out the partial penguin rather than crop, once I decide on the final target image file. Same goes for the sky and any cleanup on the right. So, I'd still like to know which version you prefer regarding the bottom, where all the effort has gone to this point.

Yes, the seal pup had moved in the short time between images 4 & 5. In retrospect, I assume I could probably have fixed it by painting on the appropriate layer mask in the stack - yes? My point, though, was that as good as PTGui is, it missed that while CS4 didn't. Of course, a little painting on a layer mask is probably faster and easier than examining the stitches from both programs each time for the "better" one.

You can see some of my completed images from the trip on the Home Page of my website.

Steve

http://www.stevegouldphotography.com

PS Is there a way on/in BPN to create a "signature" that includes my website link so I don't have to type it in each time in the future?

[Michael Gerald-Yamasaki]

Michael,
Would you point out where you think you see the seams in the CS4 stitch?

Steve,

Greetings. When I said:

- Select a crop to transform putting the blending seam in a place where cloning/merging is less obvious.

you thought I meant that I saw seams? Sorry, didn't mean to imply that I saw seams.

I like the work you did in #17. Another little trick you might try to cut down on time is:

- On a separate layer warp the image until problem area fixed.
- Use a mask & paint in the fixed area (leaving most of the image alone) This is kinda similar to cloning

Also, one of the better features of the CS4 stitch (unless they changed it from CS3, what I have)... is the ability to merge images of different sizes (crops). I've not been able to get PTGUI to do this. I've on several occasions been able to crop a portion of a photo taken outside of the pano sequence and merge it in to cover problematic areas (or to add a subject, even). I continue to use the CS3 stitcher for this feature (but PTGUI for everything else ;-).

Oh, I'm also a happy smartblend user, but I think you will find that significant movement will usually require (in PTGUI) saving layers and hand masking the problem areas. PTGUI has a tutorial for this on their site.

Cheers,

-Michael-

[Fabs Forns]

Did you capture this as horizontals? What I do is take the images as vertical so I have room to spare when the pano stitch does the white stuff.
Please disregard if that is not the case here.

[Steve Gould]

Michael,
Thanks for the clarification; I was concerned that you thought you saw seams.
Let's focus just on the bottom repair. Aside from not cropping the top or sides, yet, why do you like #17 over #18, what don't you like about #18?
I'll try your suggestion of doing the Warp repair on a separate layer and just painting in the repair with a layer mask and see if it doesn't cause obvious disconformities with the nearby background layer. I basically did the reverse of that with #19 where I used Distort and painted back what was needed from the background layer.

Fabs,
I did shoot these as verticals so I would still have some height after using such a wide horizontal area. See Roger's comments about using horizontal vs. vertical. Nothing is ever really simple. :)

Steve

http://www.stevegouldphotography.com

[Michael Gerald-Yamasaki]

Michael,
Thanks for the clarification; I was concerned that you thought you saw seams.
Let's focus just on the bottom repair. Aside from not cropping the top or sides, yet, why do you like #17 over #18, what don't you like about #18?


Steve,

Greetings. Looking at them side by side, foreground pups just have a more pleasing shape to me in #17. Nothing obvious.

Regarding the different methods... I think of them as being in my bag of tricks used as is appealing at the time. Sometimes several are needed at the same time. (er, try stitching water, ocean scenes for a challenge ;) ).

IQ (SQ - stitching quality?) is pretty hard to judge with small jpegs. The proof is in the print...

Cheers,

-Michael-

[Roger Clark]

Roger,

Thanks for you continuing responses. The crop on the left is not critical at this point. #17 is clean there relative to the others only because I haven't done the final crop other than at the bottom, but thanks for mentioning it in case I forgot to take care of that. I will clone out the partial penguin rather than crop, once I decide on the final target image file. Same goes for the sky and any cleanup on the right. So, I'd still like to know which version you prefer regarding the bottom, where all the effort has gone to this point.

(Note with the addition of another reply, it is now panel 18).
As far as the animals go and the bottom, since each version uses the same images, not much has changed. The web images are too small to see any residual artifacts from warping. So I based my view on overall presentation, including the crop.



Yes, the seal pup had moved in the short time between images 4 & 5. In retrospect, I assume I could probably have fixed it by painting on the appropriate layer mask in the stack - yes? My point, though, was that as good as PTGui is, it missed that while CS4 didn't. Of course, a little painting on a layer mask is probably faster and easier than examining the stitches from both programs each time for the "better" one.

Yes, you can fix it in the stack. You can also tell ptgui to use more control points, and you can also add control points by hand to help force a warp around moving subjects. While CS4 may have done better on this image, you'll likely find that no one software does better on all images, so it comes down to flexibility to get around problems.



You can see some of my completed images from the trip on the Home Page of my website.

Steve

http://www.stevegouldphotography.com

PS Is there a way on/in BPN to create a "signature" that includes my website link so I don't have to type it in each time in the future?

Very nice Steve. I really like the first image on your home page.

Yes, you can create a signature that includes the website. Under "quick links" go to edit "edit signature" and add it there.

Roger

[Steve Gould]

Roger,
Thanks for the additional help.
Steve