CP detector improvements - Manual CP editor

[AlexandreJ]

CP detector improvements - Manual CP editor

by AlexandreJ » Tue Nov 06, 2007 8:06 am

Following the poll launched by hank, let's start to study this future feature.
There's 2 points here :
- Improvements of SIFT cp detector : this is under heavy development and we already find a way to do that ! Now, we miss one stuff : cases studies. We have some two pictures pairs that doesn't give anything with the current system and that is doing great with the next one, but we need more such hard to stitch 2 images pairs. So if you have some, let's share them !
- Manual CP editor : what do you want ? A totally manual system like in PtGui ? Something with a little helper when positionning CP ?

[GURL]

by GURL » Tue Nov 06, 2007 1:31 pm

I don't want to manually compete with SIFT when a large number of CP per link (e.g. 50 CPs) is involved:
- I would like an easier way to instruct APP and/or SIFT to place more CPs here (e.g. skyline) and to remove any CPs there (e.g. moving clouds, tree branches on a windy day, parallax errors prone locations, etc)
- I would like an easier way to delete "long links" (e.g. corner to corner link on fisheye images or very large number of links when using braketing.)
- I would like an easier way to move unlinked (or "miss-linked") pictures manually.

Those matters having been settled, then, if it's possible to reduce the number of CP under 10 per link or possible to implement "heavy weighted manual CP"...
___________

A moins de leur donner un poids plus elevé (10 fois plus ?) je n'ai pas envie d'entrer en compétition avec SIFT même si je suis persuadé (à  tort ou à  raison, peu importe) d'avoir la patience et l'expertise nécessaire pour les placer avec une précision remarquable aux endroits les plus judicieux. (Par contre je me vois très mal en train d'expliquer comment il faut faire pour bien le faire !)

En plus, une fois en mesure d'imposer mes propres CP, je ne vois pas pourquoi je ne pourrais pas aussi imposer ma propre manière de faire pendant l'optimisation (il est un peu trop "auto" cet optimiseur, non?)

Avec PTassembler et avec PTgui (jusqu'à  il y a quelques mois c'est PTgui que j'utilisai pour assembler mes panos fisheye) il s'agit de travailler avec au maximum 10 CPs par lien et un nombre restrent de liens, autrement dit de travailler CP par CP.

Avec APP c'est une gestion beaucoup plus globale qui me semble adaptée. J'ai envie de dire que c'est comme la différence entre les 3 ou 4 chèvres qu'on promène au bout d'une chaîne comparées à  un troupeau de plusieurs centaines de moutons qu'on conduit avec des chiens de berger (:D je trouve l'image un peu trop bucolique, mais je n'en trouve pas d'autre. Peut-être faudrait il invoquer la loi des grands nombre et les calculs de probabilité...)

Que les chiens de berger d'APP aient besoin d'être mieux choisis et mieux dressés me semble évident...

Pour ce qui est de faire aussi bien que PTgui sur des photos au fisheye après la pose des deux premiers CP: c'est évidemment possible - y'a pas de raison - mais je doute que ce genre d'interface puisse être conçue en 5 minutes.

On peut imaginer un Manupano Pro en plus d'Autopano PRO (par exemple à  cause du "sic" de Johnh dans le dernier post de ce thread http://www.tawbaware.com/forum2/viewtopic.php?t=2719 mais avant je recommande vivement la lecture des aventure d'Eastman. Les pros de la photographie de l'époque (sur plaque de verre, au colodion humide, avec tente-laboratoire à  transporter su place) s'en moquaient. Eastman en a déduit qu'il devait vendre ses Brownies et ses pellicules aux femmes et aux enfants d'abord et ça lui a plutôt bien réussi, non?

[[bo]]

by [bo] » Tue Nov 06, 2007 2:39 pm

I don't want anything!

So far I've had no problems with control points, placement or detection. But I'd like to second the request "an easier way to delete long links". Similar to the "keep only CPs with RMS higher than...". We've already had a discussion on this, where GURL was talking about angular distance, etc.

I'd rather have a threaded SmartBlend or even the new APP internal smart anti-ghosting blender (dubbed by me MagicBlend) but that's another topic altogether.

[hankkarl]

by hankkarl » Tue Nov 06, 2007 3:54 pm

I like the way hugin lets you add manual CPs especially the checkbox to say "use an assistant to fine tune the points".

Perhaps an option to let the user say "a manual CP is worth xxx automatically generated CPs"

I like the idea of "auto-delete long links"

I'd like an auto way to generate CPs based on contrast or edge finding, rather than SIFT. That is, automatically add CPs based on some algorithm closer to human vision.

I like GURLs idea about keeping CPs off clouds and putting them on the horizon, one way to implement it is to show all the CPs on the main window when in the CP editor (we'd need an option "show links or show CPs".) And I'd want to be able to delete CPs on that main window so that its easier to delete CPs on clouds, water, etc. An add function could add CPs on a user-specified area (draw the box, hit the add CP button) to put CPs where you want them. -- A way to automatically remove CPs from clouds, waves and moving objects is probably more of a dream than a reality.

Another option would be for APP to give a better indication of "goodness" of the link. This can be done by taking two pictures, finding the overlap based on CPs, and doing some math on the pixels in that overlaped part. The math may be to subtract one pixel from its corresponding pixel in the other region for all the pixels, then find the average of the differences of all the pixels, and give an RMS value of all the differences from the average. APP could give the user an option to delete links where the overlap correspondence is less good. This could also shown visually by finding subtracting each pixel from its corresponding pixel to give a picture of the differences in the region, then color-coding the variation from the average difference for each pixel. This resulting map could be used as a transparent overlay on the pictures to show what items don't line up.

Last edited by hankkarl on Tue Nov 06, 2007 3:50 pm, edited 1 time in total.

[Stephmw]

by Stephmw » Tue Nov 06, 2007 6:13 pm

I agree with the hankkarl, the Hugin way of adding manual CPs is very good. An optional auto-assist is welcome but not strictly speaking necessary for me.

[DrSlony]

by DrSlony » Mon Apr 28, 2008 2:07 pm

I agree that a feature to manually place a CP in an exact spot would be useful sometimes.

Also what would be great would be an option to prioritise the correct alignment of 'narrow & long' areas. Let me elaborate:
This would be very usefull for panos with cables, tiles, wooden beams, wall edges, girders, etc. APP would add extra priority to CPs that are placed along long but thin areas, like those examples, and do its best to align those areas between shots since misalignments between those parts would be most visible.

[digipano]

by digipano » Sat May 24, 2008 7:45 pm

I have the following suggestion to improve the CP in next version.

1. It should be improved so it can detect the low contrast patterns
2. It should have some kind of automatic CP placement when adding CP manually like ptgui
3.We have to manually advance to next images by manually selecting 2 images cant we have "next" button so it brings the next 2 images in the windows.
4. separate Cancel geometric analysis button (as of now it takes quite long to cancel)
5. Hugin kind of manual CP would be good.

[DrSlony]

by DrSlony » Sat Jul 05, 2008 11:09 pm

Most often the parts of a panorama that are incorrectly aligned, and parts where incorrect alignment is most visible, are edges. I was wondering about using some form of edge detection to help properly align images. I see two advantages to using this:
1- some edge detecting algorithm, or simply a blur/contrast combination like in photoshop/gimp, find edges. Then the SIFT algorithm places a high priority on all CPs located along these edge lines.
2- the edge lines can serve as a guide as to whether the images were correctly aligned. It finds a common edge pattern in the two images and then, after stitching the two images and deciding at which exact point smartblend will chop off each image's overlap, it checks if this pattern has any sudden 'interruptions', if it has been correctly overlayed in common parts. If the edge line does suddenly stop in the stitched image but does NOT stop at this point in the source image, then a bad stitch alarm is raised.

It all is nice and easy here, and I have no experience in these kinds of things, but perhaps the idea could be used to solve some frequent problems?

Some sample screenshots of how this might work:
1 - the original image. I cut it into two images with overlap.
2 - an example of how APP could have detected CPs.
3 - the results of these CPs gives a bad stitch, especially visible in the cables.
4 - APP checks edges if there are any sudden interruptions. It finds several broken edges in places where they were not broken in the original left source photo. A bad stitch alarm is raised,
5 - APP gives priority to these places,
6 - and tries to find some CPs in these broken edge areas.
7 - reoptimized, and now the stitch is good.

Last edited by DrSlony on Sat Jul 05, 2008 11:10 pm, edited 1 time in total.

[hankkarl]

by hankkarl » Sat Apr 25, 2009 5:05 am

I agree that a feature to manually place a CP in an exact spot would be useful sometimes.

Also what would be great would be an option to prioritise the correct alignment of 'narrow & long' areas. Let me elaborate:
This would be very usefull for panos with cables, tiles, wooden beams, wall edges, girders, etc. APP would add extra priority to CPs that are placed along long but thin areas, like those examples, and do its best to align those areas between shots since misalignments between those parts would be most visible.

Perhaps APP could keep more of the points near edges with high contrast? (long, narrow features have two edges)

Or SIFT could try and put more points near the edges? Right now, the CP distribution is supposed to be random. But I think we need some CPs in each corner, some in the middle, and most CPs on high contrast edges.

Last edited by hankkarl on Sat Apr 25, 2009 5:07 am, edited 1 time in total.

[AlexandreJ]

by AlexandreJ » Sat Apr 25, 2009 8:29 am

One note. The internal project on this part has made good advances. We now have 4 kinds of CP and 2 type of descriptors.
CP types : 2 of them are edges aware meaning they will locate themself on edges first. 2 others are blob aware like SIFT : they locate themself on uniform area.
We now study how to auto switch from one type to another one. For example, is we found out that there is a lot edge aware cp and not much blob cp, then it's probably an architecture picture with a lot of lines and we'll boost the first type to be sure that edge will be continuous. On the other hand, you may not have any edge aware CP meaning you are going to stitch together clouds for example. In this case, SIFT is by far the best.
Anyway. This study is really interesting and will really change the software in the future.

[superschroeder]

by superschroeder » Thu May 14, 2009 7:01 pm

Hi,

any news on the manual control point editing capabilities? The discussion of such a feature has been ongoing since 2007 in this forum. I would vote for similar ones like PTgui provides.

I have major issues with auto-detection of control points in my fisheye images. The stiched panoramas are totally useless. I know Autopano works well on standard lenses but fisheye seems to overburden its capabilities.

Cheers

Uli

[mediavets]

by mediavets » Thu May 14, 2009 7:17 pm

Uli

any news on the manual control point editing capabilities? The discussion of such a feature has been ongoing since 2007 in this forum. I would vote for similar ones like PTgui provides.

http://www.autopano.net/forum/p45316-today-08-51-28#p45316

I have major issues with auto-detection of control points in my fisheye images. The stiched panoramas are totally useless. I know Autopano works well on standard lenses but fisheye seems to overburden its capabilities.

Really? - the latest version of APG V2 has done a superb job with every fisheye image set I've tested, all sorts of FE lenses and a whole variety of pano scenes, daytime, nightime, interiors, exteriors...

Show us some screenshots of where it has failed for you with fisheye images.

Last edited by mediavets on Thu May 14, 2009 7:19 pm, edited 1 time in total.

[neweramike]

by neweramike » Thu Sep 03, 2009 5:52 pm

Yes Please add a manual method of adding control points.. why?
My work is hand held, linear panorama, using 24mm lens in a high parallax environment..
Auto pano 203 and 147 are fantastic but where there are areas that it can not recognize, I can not add points in. If the focus plane is a little off of the background, such as with a fish floating infront of the background, I need to FORCE control points in.

Can this be done with an export to PTgui and reimport to Autopano?

[mmokrejs]

by mmokrejs » Wed Dec 30, 2009 9:54 pm

One note. The internal project on this part has made good advances. We now have 4 kinds of CP and 2 type of descriptors.
CP types : 2 of them are edges aware meaning they will locate themself on edges first. 2 others are blob aware like SIFT : they locate themself on uniform area.
We now study how to auto switch from one type to another one. For example, is we found out that there is a lot edge aware cp and not much blob cp, then it's probably an architecture picture with a lot of lines and we'll boost the first type to be sure that edge will be continuous. On the other hand, you may not have any edge aware CP meaning you are going to stitch together clouds for example. In this case, SIFT is by far the best.
Anyway. This study is really interesting and will really change the software in the future.

I think what you need in addition is to associate some error probability value with each CP. For example, to reflect probability that a CP in the big dark sport should not lie few pixels aside. Often, my hand held pictures have slightly different horizon and that I hardly fix by manual placement of CPs even into corners.

Currently there is no way to re-position some CPs by local fine-tuning. For example, in a cloud of "green" CPs there are few in the middle which are "brown". Those were clearly initially misplaced. Let's say I do not want to drop them but definitely APP could shift them few pixels aside within the cloud of CPs.

[manuel rosario]

by manuel rosario » Fri Feb 26, 2010 4:53 pm

Hi,

I am taking images on some interior shots (churches and houses) as my example as I am new to Autopano Giga which I recently registered. The stitches are almost perfect except for quite a few misalligned structural members, which sometimes are not pleasing to the eyes. The proposed manual adding of cp (similar to ptqui) will help me best.

Cheers
Manny

[GURL]

by GURL » Fri Feb 26, 2010 9:01 pm

I have major issues with auto-detection of control points in my fisheye images. The stiched panoramas are totally useless. I know Autopano works well on standard lenses but fisheye seems to overburden its capabilities.

I often find spherical panos being much easier to optimise when what I would call "useless links" are removed (manually, alas) next to zenith and less often next to nadir.

I have my own theory about that :

when only a limited area is overlapped between two source images being near the zenith (or encompassing the zenith):
1) the CP are often located next to an image top corner, a dubious location for CP
2) they all are in a small area
...as a result the optimizer is able to reach for this particular link a low (but wrong) RMS by making small changes in yaw, pitch, roll (*) and/or by making small changes in lens distortion correction.

About point (2): when the links disposition is:
[1]---a---[2]---b---[3]
\________c_______/

as I recently observed in two spherical panos , these "skip over" links (like c) tends to get a much better RMS than links like a and b. In such cases I remove the "skip over" links and as a rule this improve the stitch.

It should be noted that in a spherical pano it's often difficult or impossible to avoid groups of images overlapping each of the other images in this group. This is especially true when no less than 5 fisheyes images in the "top row" (and more than 5 when a rectilinear lens is used) are going past the nadir and reach the other side (dépassent le zénith)

As you can see in the following 24 mm equivalent non-fisheye example, I didn't removed many links but removing a few of them really helped ( there are only 2 zenith shots, though the rectilinear lens I used is not a wide-angle one, because PapySpheric designed the preset this way...)
___
I removed the remark I first put there

Last edited by GURL on Sun Feb 28, 2010 9:26 pm, edited 1 time in total.

[GURL]

by GURL » Sat Feb 27, 2010 2:59 pm

As a proof that "the larger the amount of links the better" would be a wrong rule, removing more links enabled to reach a point where any link have a lower than 2.0 RMS.

[hankkarl]

by hankkarl » Sat Feb 27, 2010 6:47 pm

One note. The internal project on this part has made good advances. We now have 4 kinds of CP and 2 type of descriptors.
CP types : 2 of them are edges aware meaning they will locate themself on edges first. 2 others are blob aware like SIFT : they locate themself on uniform area.
We now study how to auto switch from one type to another one. For example, is we found out that there is a lot edge aware cp and not much blob cp, then it's probably an architecture picture with a lot of lines and we'll boost the first type to be sure that edge will be continuous. On the other hand, you may not have any edge aware CP meaning you are going to stitch together clouds for example. In this case, SIFT is by far the best.
Anyway. This study is really interesting and will really change the software in the future.

OK, but the two types are not mutually exclusive--you may want to keep some blob CPs that are near an edge CP. etc. It may be better to prefer some CPs that are distributed evenly in the image area taken by the "sweet spot" of the lens, and also prefer some CPs near edges. This doesn't necessarily mean take half the CPs from an even distribution and half from edges.

Also, GURL made some observations:

1. automatically get rid of "skip over" links, or at least use the length of a link to weight it.
2. CPs in corners shouldn't count as much.
a. Looking at MFT charts, rectilinear lenses still have a circular "sweet spot".
b. A mask for rectangular images like the fisheye mask would eliminate the CPs at the bad part of the lens.
c. The mask may be a gradient mask, where each CP is weighted depending on its distance from the center of the lens.
d. Lens correction may be accounted for, so the mask could be automatic and vary for each lens, and also vary by f/stop used. Or the lens distortion algorithm could be used to weight each CP depending on the amount of distortion at that point.

We also need a better way than RMS to determine the goodness of a pano. it could be simply subtracting one pixel from the overlaping pixel, and giving us both the average and the standard deviation of this for all the pixels in an image. APP could also show this as a picture which shows the difference (in luminance? in RGB?) between each pixel as a greyscale picture. APP could then average the pixels, use that average as black and show the difference as greyscale, so errors become more obvious..

Since most panos have regions with more than two images contributing to a pixel, the math gets harder--but it may be just taking the average values of the pixels, and summing the absolute value of the differences from average.

[Tipjip]

by Tipjip » Sat Apr 10, 2010 10:43 pm

One note. The internal project on this part has made good advances. [...]
Anyway. This study is really interesting and will really change the software in the future.

Is there any date for a beta with this functionality ? I read in another thread that this is supposed to be part of 2.1. Is there a (approximate) date for that or a roadmap?

Christian

[manuel rosario]

by manuel rosario » Fri May 21, 2010 3:01 am

i hope, the totally manual cp editor like ptgui could be included in the upcoming 2.5 version. I am still getting mis allignment on interiors of houses on v2.0.7. I am using 12mm of the sigma 12-24 lens and having 8+8+nadir+zenith in a 360/180 virtual tour. This is my only problem, everything else is working fine with me. I am using mac os 10.6.3. Thank you.

Manny

[bradtem]

by bradtem » Wed Jun 02, 2010 4:50 am

I also want this, as well as hand control of blending regions. APP is great at automatically assembling many panos, and we all love when it can do this and we don't have to do any work to get a good pano. But the reality is that sometimes it just can't do it. Sometimes due to its own limitations, and sometimes due to limitations of the shots. Hand drawn CPs and hand drawn blends are a pain to do, but they are better than simply not being able to get a good pano, and learning that after coming home from somewhere thousands of miles away!

In particular, there are just times when the human eye can see the overlap clearly, but APP can't see it, no matter how you draw the boxes when telling it to add CPs. I don't know why, and of course it is good to try to make it better so this happens less and less. But as long as it can still happen we need the manual CPs. Sometimes the manual CPs will even be cheating, where the human eye can't see the CP but the brain can -- for example in sea and sky or other moving things.

Likewise hand-drawn blends.

[Christian Stüben]

by Christian Stüben » Tue Jul 10, 2012 7:22 am

- Manual CP editor : what do you want ? A totally manual system like in PtGui ? Something with a little helper when positionning CP ?

Hmm, beside a better automatic recognition that is *always* desired, i would like to see a manual point editing like in ptgui. The syncing between the two tiles is simply great when manually editing and adding new points. It improves your workflow just incredible.

You can improve automatic recognition as you will, there are always some exceptions left, like regular structures, great uniformly colored areas, that will oppose a good automativ recognition.

(I already have done a feature request for this)

greetings from germany
Chris

[AlexandreJ]

by AlexandreJ » Tue Jul 10, 2012 9:34 am

You already have that is Autopano. It is working like in PtGui in fact, with a little windows that highlight the details of the zone. You can move points, etc. Easy.

[toto-foto]

Re: CP detector improvements - Manual CP editor

by toto-foto » Fri Aug 08, 2014 11:36 am

I would be happy if there is a "rotate" tool in CP editor. When stitching Live Pano, there is horizontal video frame and vertical photo.

CP Editor

If I can rotate one of them to have the same orientation (like in PTGui), it would be useful and no more need to rotate my head %)