The geometry of rank drop in a class of face-splitting matrix products: Part I

1 Introduction

Let ⊗ denote the Kronecker product; see [30]. We are interested in solving the following problem:

In signal processing, the matrix Z_k is known as the face-splitting product [27] of the two matrices X_k ∈ ℝ^k×3 with rows x1⊤,…,xk⊤ and Y_k ∈ ℝ^k×3 with rows y1⊤,…,yk⊤. The face-splitting product is a special case of the Khatri–Rao matrix product in linear algebra and statistics [22]. However, our interest in Problem 1.1 comes from an estimation problem in 3D computer vision known as the 7-point problem [18].

In 3D computer vision the world (or the scene) is modeled as a collection of points in ℙ³. Pinhole cameras imaging the scene are modeled as full rank linear projections from ℙ³ to ℙ².

Let P₁, …, P₇ ∈ ℙ³ be a scene consisting of seven world points, and let A₁ and A₂ be two pinhole cameras such that they map the scene to images points x₁, …, x₇ and y₁, …, y₇ respectively (here we assume that the images are finite points in ℙ², so they can be de-homogenized to points in ℝ²). A solution F to the 7-point problem that has rank 2 is known as a fundamental matrix. The fundamental matrix determines the pair of cameras A₁ and A₂ up to a homography of ℙ³; see [18; 23].

Observe that the linear constraints in Problem 1.2can be re-written as ^[1]

where vec(F) is the 9-dimensional vector obtained by concatenating the columns of F. Thus solving the 7-point problem requires computing the intersection of the null space of a 7 × 9 face-splitting product with a determinantal variety. Generically it will have three distinct solutions, one of which will always be real. When this is not the case, i.e., when Problem 1.2has less than three distinct solutions or has an infinite number of solutions, it is said to be ill-posed [10].

Besides being interesting on its own as a mathematical phenomenon, ill-posedness is of practical interest because of a meta-theorem in numerical analysis that says that the numerical conditioning of a problem is inversely related to the distance to the nearest ill-posed problem [9; 8; 5]. So, understanding the data for which Problem 1.2is ill-posed also helps us understand how and when it is numerically hard to solve it using finite precision arithmetic.

There are two ways in which Problem 1.2can have an infinite number of solutions. The first is that the null space of Z₇ has dimension greater than 2. The second is that the null space even though it is 2-dimensional, lies entirely inside the determinantal variety. So, a sufficient condition for ill-posedness is that the corresponding face-splitting product matrix be rank deficient and thus our interest in Problem 1.1.

A first answer to Problem 1.1 is that rank(Z_k) < k if and only if all the maximal minors of Z_k are zero. These minors are bi-homogeneous polynomials in x_i and y_i, and typically do not shed much light on the geometry of the point configurations {x_i} and {y_i} that cause Z_k to drop rank. Our goal is to characterize the rank deficiency of Z_k in terms of the geometry of {x_i} and {y_i}.

To this end, we begin by observing that the rank of Z_k is a projective invariant. First, note that multiplying the x_i’s and y_i’s by non-zero scalars scales the rows of Z_k and does not change its rank. Second, if H₁, H₂ are invertible matrices, then from the mixed-product property of Kronecker products we have that

The matrix H1⊤⊗H2⊤ has full rank since H₁ and H₂ are invertible, and hence rank⁡(Zk′)=rank⁡(Zk). These observations imply that rank(Z_k) is independent of scaling and choice of coordinates in ℝ³ × ℝ³. Therefore, we should be studying the problem over projective space. While our computer vision applications care about real input, mathematically there is no loss in assuming that the input points (x_i, y_i) are complex because ℝ ⊂ ℂ. Therefore, letting ℙ denote projective space over ℂ we pass to the following formulation of Problem 1.1 which will be the main subject of study in this paper.

Let Z_k{l} denote a submatrix of Z_k with l of the k rows. If rank(Z_k{l}) < l, then rank(Z_k) < k, and any condition on the l points (x_i, y_i) that contribute the rows of Z_k{l} and cause Z_k{l} to drop rank will be a condition on {(x_i, y_i), i = 1, …, k} such that Z_k drops rank. We will call such a condition an inherited condition for rank drop. For each value of k we will be most interested in the non-inherited conditions on the input that make Z_k rank deficient.

As an illustration of the type of answers we will provide, consider the version of Problem 1.3 in ℙ¹ × ℙ¹. Suppose {x_i} and {y_i} are each sets of distinct points in ℙ¹. In Theorem 4.1 we prove that the 4 × 4 matrix Z₄ is rank deficient if and only if the cross-ratio of x₁, x₂, x₃, x₄ equals that of y₁, y₂, y₃, y₄. The cross-ratio is the fundamental projective invariant of four points in ℙ¹.

It is a classical fact that the geometry of a finite set of points in ℙ², that remains invariant under the action of PGL(3), can be expressed via polynomials in the 3 × 3 determinants (brackets) of the matrices whose rows are the points. Many of our answers to Problem 1.3 will be phrased in terms of these bracket expressions. The results have analogs in terms of cross-ratios when the points are in general position.

We solve Problem 1.3 across this paper (Part I) and its sequel (Part II, see [7]). In this paper we address the Cases 2 ≤ k ≤ 6 and in the next we address k = 7, 8, 9. A great deal of classical algebraic geometry emerges starting with k = 6. This motivated the split since k = 6 is rather elaborate and we wanted to keep the papers to a reasonable length. Also, the flavors of results in the two papers are slightly different. For k ≤ 6, we are able to obtain both geometric and algebraic characterizations of rank drop without any assumptions on the points. For k ≥ 7, the tools and results become more geometric and require a type of genericity of the input.

2 Preprocessing the input

We begin by showing that one can make simplifying assumptions on the input to Problem 1.3 which is a collection of k ≤ 6 points (x_i, y_i) ∈ ℙ² × ℙ².

We sometimes refer to (x_i, y_i) as a point pair, especially when we need to decouple the components and think of the configurations x_i and y_i separately. It will also be convenient to refer to the ℙ² containing {x_i} as Px2 and the ℙ² containing {y_i} as Py2 . Let PGL(3) be the group of invertible 3 × 3 matrices over ℂ up to scaling. An element H ∈ PGL(3) induces a homography (projective isomorphism) on ℙ².

In the introduction we informally showed that the rank of Z_k = (xi⊤⊗yi⊤)i=1k is invariant under PGL(3) × PGL(3) acting on Px2 × Py2 by left multiplication. This allows us to change bases in Px2 and Py2 independently. We state the result formally below:

Lemma 2.1 allows us to make certain assumptions on the configurations {x_i} and {y_i} which we phrase as corollaries to the lemma. While our proofs do not need these assumptions, they become critical for computations in Macaulay2. The first assumption is that the x_i’s and y_i’s in the input to Problem 1.3 are finite points. With a view towards computations, in the following lemma and in several parts of the paper, we fix representatives of points in ℙ² and write them out as vectors in ℂ³.

The other assumption we can make is that sufficiently generic configurations allow some of the points to be fixed, which will prove useful for computations.

At certain times, for the sake of symmetry, we will instead fix the first four points as

Since the two fixings are related by a homography, results from one fixing transfer to the other.

3 Cross-ratios and brackets

By Lemma 2.1, the rank deficiency of Z_k is invariant under the action of PGL(3) × PGL(3) on ℙ² × ℙ² given by (H₁, H₂) ⋅ (x, y) ↦ (H₁x, H₂y). Therefore, the input to Problem 1.3 can be thought of as the PGL(3)-orbits of k points in each ℙ². The intrinsic geometry of k points p₁, …, p_k ∈ ℙ² are the properties of the configuration that remain intact/invariant under the action of PGL(3). Since an element of PGL(3) can be assumed to have determinant 1, the above geometry is precisely the invariant properties, under the action of SL(3), of the configuration obtained by replacing each p_i ∈ ℙ² with a representative in ℂ³. We now introduce the basic tools from the invariant theory of point sets needed to formulate our results. There are numerous references for this material such as [6], [11], [12], [13] and [29].

Let ℂ[p] := ℂ[p_i1, p_i2, p_i3, i = 1, …, k] be the polynomial ring in the variables p_ij for i = 1, …, k, j = 1, 2, 3, that represent the coordinates of k points p₁, …, p_k in ℙ² written via representatives in ℂ³. A polynomial f ∈ ℂ[p] is invariant under SL(3) if f(p₁, …, p_k) = f(Hp₁, …, Hp_k) for all H ∈ SL(3). It is a well-known classical result that the invariant ring ℂ[p]^SL(3), which is the collection of all polynomials in ℂ[p] that are invariant under SL(3), is generated by the 3 × 3 minors of the matrix whose rows are the symbolic p1⊤,…,pk⊤. In other words, ℂ[p]^SL(3) is generated as an algebra by the bracket polynomials

for all distinct choices of i, j, l ∈ {1, …, k}. In our situation, we will write [ijl]_x (respectively, [ijl]_y) when the input points come from {x_i} (respectively, {y_i}). Equations in brackets can be used to express geometry. For example, three points p_i, p_j, p_l ∈ ℙ² are collinear if and only if [ijl] = 0, and 6 points lie on a conic if and only if they satisfy the bracket equation (12) in Lemma 3.5.

Often we consider points in a line ℓ in ℙ² as points in ℙ¹ where we can compute the bracket [ij] for p_i, p_j ∈ ℓ by identifying ℓ with the line parameterized as say (*, *, 0), and then dropping the 0 coordinate to get

When we have many points p₁, …, p_k on a line and a point u not on that line, the ℙ² brackets [iju] will be closely related to the ℙ¹ brackets [ij].

The cross-ratio is the only projective invariant of 4 points in ℙ¹ in the following sense.

Permuting the points on the line changes the cross-ratio in a systematic way, i.e., if

then

for all σ ∈ S₄. Thus we do not need to consider multiple orderings. The cross-ratio is 0, 1, ∞, or 00 if and only if the points are not distinct. See [26, III, § 4–5].

In Example 1, the points (x_i, y_i) lie on the variety defined by the bracket equation (10) but do not satisfy the cross ratio equality (9). The discrepancy is because cross-ratios are well-defined only on an open set, while the corresponding bracket equation holds on the Zariski closure of this open set. Since our interest is in describing the algebraic variety of input points that make Z₄ rank deficient, the bracket expressions are the correct choice. However, whenever the points are in sufficiently general position, we can pass to cross-ratio expressions which, besides being elegant, have the advantage that they can be visualized easily.

We will also need planar and conic cross-ratios which we now define.

Note that planar cross-ratios are preserved under a homography of ℙ². For 5 distinct points, the cross-ratio around x₅ can be obtained geometrically by drawing the 4 lines p_ip₅ for i = 1, 2, 3, 4, cutting them with a transversal, and then computing the cross-ratio of the intersection points. See Figure 1. Also, the planar cross-ratio is transformed by permutations of p₁, …, p₄ similarly to the line cross-ratio.

Figure 1

For distinct points p₁, …, p₅ ∈ ℙ², the planar cross-ratio (p₁, p₂; p₃, p₄; p₅) equals the 4 point cross-ratio (a₁, a₂; a₃, a₄).

Lemma 3.5 makes the following definition of a conic cross-ratio well-defined.

Any non-degenerate conic ω is isomorphic to ℙ¹. If ι : ω → ℙ¹ is one such isomorphism then

4 The cases k ≤ 4

We will now answer Problem 1.3 in the cases k ≤ 4. As a warm up to Problem 1.3 we first consider the analogous question in ℙ¹ × ℙ¹. Recall that any condition on a proper subset of {(x_i, y_i)} that makes the corresponding rows of Z_k dependent is called an inherited condition for the rank drop of Z_k.

We now consider Problem 1.3 for k ≤ 4 points (x_i, y_i) ∈ ℙ² × ℙ². Again all of the statements can be checked in Macaulay2 by following the same general strategy as in the proof of Theorem 4.1.

5 The case k = 5

We now consider the rank deficiency of Z₅. In this case, we present two theorems. Theorem 5.1 is a characterization of rank deficiency in terms of brackets. We will see that a necessary condition for rank drop is that one set of points (say {y_i}) is on a line ℓ. Theorem 5.2 gives a geometric characterization of rank deficiency when the points are sufficiently generic, i.e., when the y_i’s on the line ℓ are distinct and the x_i’s are in general position. In this case the rank drop of Z₅ is equivalent to the existence of an isomorphism from the conic ω through the x_i’s to the line ℓ taking x_i to y_i. This is analogous to Theorem 4.2 (3)(b) where there is a homography taking x_i to y_i under sufficient genericity. We will illustrate both results using Example 2.

We now formalize the second half of Example 2 as a theorem. As seen in Theorem 5.1, for Z₅ to drop rank, either the x_i’s or the y_i’s have to be on a line. Suppose the y_i’s are distinct and on a line, and the x_i’s are in general position. Then the x_i’s lie on a conic ω and we will see that there is an isomorphism taking ω to the line containing the y_i’s so that each x_i maps to y_i.

Proof

To prove that (1) implies (2), let c ∈ ω ∖ {x₁, …, x₅}. Then for all distinct i₁, i₂, i₃, i₄, j ∈ {1, …, 5}

(yi1,yi2;yi3,yi4)=(xi1,xi2;xi3,xi4;xj)=(xi1,xi2;xi3,xi4)ω=(xi1,xi2;xi3,xi4;c) (25)

for any c ∈ ω. The first equation is from (20) in Theorem 5.1 and the other two equalities follow by Definition 3.6. Let cx_i be the line through c and x_i and let ℓ be any line such that c ∉ ℓ. Then if we define a_i to be the unique intersection of ℓ and cx_i it follows that

(ai1,ai2;ai3,ai4)=(xi1,xi2;xi3,xi4;c)=(yi1,yi2;yi3,yi4) (26)

for all i₁, i₂, i₃, i₄ ∈ {1, …, 5}. See Figure 2. It follows that there exists homography H such that Ha_i ∼ y_i for all i. Therefore if Tc′ is the projection in Px2 centered at c onto the line ℓ, then T_c := H Tc′ is the desired projective transformation with T_c(x_i) ∼ y_i for all i.

Figure 2

We construct the projection with center c.

Clearly (2) implies (3). To show that (3) implies (1), suppose that such a projection exists and is centered at c. Then

(xi1,xi2;xi3,xi4;xj)=(xi1,xi2;xi3,xi4)ω=(xi1,xi2;xi3,xi4;c)=(yi1,yi2;yi3,yi4) (27)

for all distinct i₁, i₂, i₃, i₄, j ∈ {1, …, 5}. The first two equalities follow by Definition 3.6 and the last equality follows by hypothesis. It immediately follows that Z₅ is rank deficient by Theorem 5.1.

To show that (1) implies (4) we may assume that the cross-ratio equalities (20) hold. Then define an isomorphism F : ω → ℓ_y by F(x₁) = y₁, F(x₂) = y₂, F(x₃) = y₃. Then for all p ∈ ω ∖ {x₁, x₂, x₃}, F(p) = q is such that (x₁, x₂; x₃, p)_ω = (y₁, y₂; y₃, q). It then follows that F(x₄) = y₄ and F(x₅) = y₅ because

(x1,x2;x3,x4)ω=(y1,y2;y3,y4)(x1,x2;x3,x5)ω=(y1,y2,y3;y5) (28)

by hypothesis (Theorem 5.1).

Finally, to show that (4) implies (1), suppose that such an isomorphism F exists. Then

(xi1,xi2;xi3,xi4;xj)=(xi1,xi2;xi3,xi4)ω=(F(xi1),F(xi2);F(xi3),F(xi4))=(yi1,yi2;yi3,yi4) (29)

for all i₁, i₂, i₃, i₄, j ∈ {1, …, 5}. The first equality follows by Definition 3.6 and the second and third equalities follow by hypothesis. It then follows that Z₅ is rank deficient by Theorem 5.1.□

6 The case k = 6

The final case we will consider in this paper is a set of 6 inputs {(xi,yi)}i=16∈P2×P2. We organize our results in two parts. In the first part we state algebraic characterizations of the rank deficiency of Z₆. These are analogous to the results in the previous sections in that the results are in terms of invariant theory. In the second part, we explain the geometry that leads to these algebraic statements. This relies on the classical theory of cubic surfaces and their combinatorics. The geometry works in an open region where the input is sufficiently generic. However, as we saw in previous sections, the algebra that it translates to describes the Zariski closure of these open regions and hence the set of all inputs that make Z₆ rank deficient.

We say that p₁, …, p₆ ∈ ℙ² are in general position if at most 2 points are on a line and at most 5 points are on a conic. In several results below we need that the x points and/or the y points are in general position. We will also need that a collection of point pairs {(x_i, y_i)} ⊂ ℙ² × ℙ² is in general position, by which we mean that

1. the x points and y points are in general position, and

2. no more than 4 pairs (x_i, y_i) are related by a homography Hx_i = y_i.

Note that if (1) holds then Z₅ has full rank because all of our previous conditions for rank drop require at least the x_i’s or the y_i’s to be on a line.

The variety ℙ² × ℙ² can be embedded in ℙ⁸ as the image of the Segre map

In what follows it will be helpful to identify ℙ⁸ with the set of all complex 3 × 3 matrices up to scaling via the bijection M ↔ vec(M). Then the Segre embedding of ℙ² × ℙ² in ℙ⁸ (which we also call ℙ² × ℙ²) can be identified with the set of rank one 3 × 3 matrices up to scaling since

The Segre variety ℙ² × ℙ² ⊂ ℙ⁸ has dimension 4 and degree 6. It is the zero locus of the determinantal ideal generated by the 2 × 2 minors of a symbolic 3 × 3 matrix whose entries denote the coordinates of ℙ⁸.

We first show that given 5 points {(xi,yi)}i=15 in general position, there exists a unique 6th point (x₆, y₆) ∈ ℙ² × ℙ² such that {(xi,yi)}i=16 is rank deficient.

The existence of, and constructions for, this 6th point were noted in both [25] and [31], but not the rational formula. The rational formula shows that if (xi,yi)i=15 are real then (x₆, y₆) will be real as well. Note that if {xi}i=15 and {yi}i=15 were related by homography then the denominators in (31) would all be 0. Here is an illustration of Lemma 6.1.

In Lemma 6.16 we will prove a counterpart to the above result, where we instead begin with x₁, …, x₆ and calculate (up to homography) the points y₁, …, y₆ that will result in rank deficiency.