Affine shape adaptation

The interest points obtained from the scale-adapted Laplacian blob detector or the multi-scale Harris corner detector with automatic scale selection are invariant to translations, rotations and uniform rescalings in the spatial domain. The images that constitute the input to a computer vision system are, however, also subject to perspective distortions. To obtain interest points that are more robust to perspective transformations, a natural approach is to devise a feature detector that is invariant to affine transformations.

Affine invariance can be accomplished from measurements of the same multi-scale windowed second moment matrix ${\displaystyle \mu }$ as is used in the multi-scale Harris operator provided that we extend the regular scale space concept obtained by convolution with rotationally symmetric Gaussian kernels to an affine Gaussian scale-space obtained by shape-adapted Gaussian kernels (Lindeberg 1994, section 15.3; Lindeberg & Garding 1997). For a two-dimensional image ${\displaystyle I}$, let ${\displaystyle {\bar {x}}=(x,y)^{T}}$ and let ${\displaystyle \Sigma _{t}}$ be a positive definite 2×2 matrix. Then, a non-uniform Gaussian kernel can be defined as

${\displaystyle g({\bar {x}};\Sigma )={\frac {1}{2\pi {\sqrt {\operatorname {det} \Sigma _{t}}}}}e^{-{\bar {x}}\Sigma _{t}^{-1}{\bar {x}}/2}}$

and given any input image ${\displaystyle I_{L}}$ the affine Gaussian scale-space is the three-parameter scale-space defined as

${\displaystyle L({\bar {x}};\Sigma _{t})=\int _{\bar {xi}}I_{L}(x-\xi )\,g({\bar {\xi }};\Sigma _{t})\,d{\bar {\xi }}.}$

Next, introduce an affine transformation ${\displaystyle \eta =B\xi }$ where ${\displaystyle B}$ is a 2×2-matrix, and define a transformed image ${\displaystyle I_{R}}$ as

${\displaystyle I_{L}({\bar {\xi }})=I_{R}({\bar {\eta }})}$.

Then, the affine scale-space representations ${\displaystyle L}$ and ${\displaystyle R}$ of ${\displaystyle I_{L}}$ and ${\displaystyle I_{R}}$, respectively, are related according to

${\displaystyle L({\bar {\xi }},\Sigma _{L})=R({\bar {\eta }},\Sigma _{R})}$

provided that the affine shape matrices ${\displaystyle \Sigma _{L}}$ and ${\displaystyle \Sigma _{R}}$ are related according to

${\displaystyle \Sigma _{R}=B\Sigma _{L}B^{T}}$.

Disregarding mathematical details, which unfortunately become somewhat technical if one aims at a precise description of what is going on, the important message is that the affine Gaussian scale-space is closed under affine transformations.

If we, given the notation ${\displaystyle \nabla L=(L_{x},L_{y})^{T}}$ as well as local shape matrix ${\displaystyle \Sigma _{t}}$ and an integration shape matrix ${\displaystyle \Sigma _{s}}$, introduce an affine-adapted multi-scale second-moment matrix according to

${\displaystyle \mu _{L}({\bar {x}};\Sigma _{t},\Sigma _{s})=g({\bar {x}}-{\bar {\xi }};\Sigma _{s})\,\left(\nabla _{L}({\bar {\xi }};\Sigma _{t})\nabla _{L}^{T}({\bar {\xi }};\Sigma _{t})\right)}$

it can be shown that under any affine transformation ${\displaystyle {\bar {q}}=B{\bar {p}}}$ the affine-adapted multi-scale second-moment matrix transforms according to

${\displaystyle \mu _{L}({\bar {p}};\Sigma _{t},\Sigma _{s})=B^{T}\mu _{R}({\bar {q}};B\Sigma _{t}B^{T},B\Sigma _{s}B^{T})B}$.

Again, disregarding somewhat messy technical details, the important message here is that given a correspondence between the image points ${\displaystyle {\bar {p}}}$ and ${\displaystyle {\bar {q}}}$, the affine transformation ${\displaystyle B}$ can be estimated from measurements of the multi-scale second-moment matrices ${\displaystyle \mu _{L}}$ and ${\displaystyle \mu _{R}}$ in the two domains.

An important consequence of this study is that if we can find an affine transformation ${\displaystyle B}$ such that ${\displaystyle \mu _{R}}$ is a constant times the unit matrix, then we obtain a fixed-point that is invariant to affine transformations (Lindeberg 1994, section 15.4; Lindeberg & Garding 1997). For the purpose of practical implementation, this property can often be reached by in either of two main ways. The first approach is based on transformations of the smoothing filters and consists of:

• estimating the second-moment matrix ${\displaystyle \mu }$ in the image domain,
• determining a new adapted smoothing kernel with covariance matrix proportional to ${\displaystyle \mu ^{-1}}$,
• smoothing the original image by the shape-adapted smoothing kernel, and
• repeating this operation until the difference between two successive second-moment matrices is sufficiently small.

The second approach is based on warpings in the image domain and implies:

• estimating ${\displaystyle \mu }$ in the image domain,
• estimating a local affine transformation proportional to ${\displaystyle {\hat {B}}=\mu ^{1/2}}$ where ${\displaystyle \mu ^{1/2}}$ denotes the square root matrix of ${\displaystyle \mu }$,
• warping the input image by the affine transformation ${\displaystyle {\hat {B}}^{-1}}$ and
• repeating this operation until ${\displaystyle \mu }$ is sufficiently close to a constant times the unit matrix.

This overall process is referred to as affine shape adaptation (Lindeberg & Garding 1997; Baumberg 2000; Mikolajczyk & Schmid 2004; Tuytelaars & van Gool 2004; Ravela 2004; Lindeberg 2008). In the ideal continuous case, the two approaches are mathematically equivalent. In practical implementations, however, the first filter-based approach is usually more accurate in the presence of noise while the second warping-based approach is usually faster.

In practice, the affine shape adaptation process described here is often combined with interest point detection automatic scale selection as described in the articles on blob detection and corner detection, to obtain interest points that are invariant to the full affine group, including scale changes. Besides the commonly used multi-scale Harris operator, this affine shape adaptation can also be applied to other types of interest point operators such as the Laplacian/Difference of Gaussian blob operator and the determinant of the Hessian (Lindeberg 2008). Affine shape adaptation can also be used for affine invariant texture recognition and affine invariant texture segmentation.

Closely related to the notion of affine shape adaptation is the notion of affine normalization, which defines an affine invariant reference frame as further described in Lindeberg (2013a,b, 2021:Appendix I.3), such that any image measurement performed in the affine invariant reference frame is affine invariant.

• Blob detection
• Corner detection
• Gaussian function
• Harris affine region detector
• Hessian affine region detector
• Scale space

• Baumberg, A. (2000). "Reliable feature matching across widely separated views". Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. pp. I:1774–1781. doi:10.1109/CVPR.2000.855899.
• Lindeberg, T. (1994). Scale-Space Theory in Computer Vision. Springer. ISBN 0-7923-9418-6.
• Lindeberg, T.; Garding, J. (1997). "Shape-adapted smoothing in estimation of 3-D depth cues from affine distortions of local 2-D structure". Image and Vision Computing. 15 (6): 415–434. doi:10.1016/S0262-8856(97)01144-X.
• Lindeberg, T. (2008). "Scale-space". Encyclopedia of Computer Science and Engineering (Benjamin Wah, ed), John Wiley and Sons. Vol. IV. pp. 2495–2504. doi:10.1002/9780470050118.ecse609. ISBN 978-0470050118.
• Lindeberg, T. (2013a). "Invariance of visual operations at the level of receptive fields". PLOS ONE. 8 (7): e66990:1–33. arXiv:1210.0754. Bibcode:2013PLoSO...866990L. doi:10.1371/journal.pone.0066990. PMC 3716821. PMID 23894283.
• Lindeberg, T. (2013b). "Generalized axiomatic scale-space theory". Advances in Imaging and Electron Physics. 178 (7): 1–96. doi:10.1016/B978-0-12-407701-0.00001-7. ISBN 9780124077010.
• Lindeberg, T. (2021). "Normative theory of visual receptive fields". Heliyon. 7 (1): e05897. doi:10.1016/j.heliyon.2021.e05897. PMC 7820928. PMID 33521348.
• Mikolajczyk, K.; Schmid, C. (2004). "Scale and affine invariant interest point detectors" (PDF). International Journal of Computer Vision. 60 (1): 63–86. doi:10.1023/B:VISI.0000027790.02288.f2. S2CID 1704741. Integration of the multi-scale Harris operator with the methodology for automatic scale selection as well as with affine shape adaptation.
• Tuytelaars, T.; van Gool, L. (2004). "Matching Widely Separated Views Based on Affine Invariant Regions" (PDF). International Journal of Computer Vision. 59 (1): 63–86. doi:10.1023/B:VISI.0000020671.28016.e8. S2CID 5107897. Archived from the original (PDF) on 2010-06-12.
• Ravela, S. (2004). "Shaping receptive fields for affine invariance". Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. Vol. 2. pp. 725–730. doi:10.1109/CVPR.2004.1315236. ISBN 0-7695-2158-4.