Quora noscript

15.3 Inner product spacesPIC

Key points

  • An inner product (15.134) is a special function that induces a rich geometry on a vector space (15.1)-(15.2), including length (15.147), distance (15.154) and angle (15.163).
  • The inner product leads to the notion of orthogonality (15.169), which in turn leads to orthgonal projection or best prediction (15.192).

In this section we review basic notions of geometry that hold for inner product spaces [W].

An inner product space is a vector space (15.1)-(15.2) with an associated inner product . An inner product is any function that takes as input any two elements in a vector space and outputs a real number, i.e.

(15.134)

To be an inner product, (15.134) must display the following properties for any vectors and any scalar :

1.
Symmetry
(15.135)
2.
Linearity
(15.136)
(15.137)
3.
Positive definiteness
(15.138)

A commonly used inner product on the vector space is the dot product [W], which is defined as

(15.139)

where recall vectors in are commonly denoted using the bold notation . We want to stress that the dot product (15.139) is one specific inner product on , but there are in fact infinitely many inner products which can be defined on , and more generally in any vector space, see Section 17.3.5 for an example of an inner product in . The only requirement for an inner product is that it needs to satisfy (15.135)-(15.136)-(15.137)-(15.138).

PICExample 15.29. Dot product.
We continue from Example 15.12. Consider the vectors (15.25)-(15.26). The dot product of and , (15.139) is calculated as

(15.140)

We can generalize the dot product to the Mahalanobis inner product

(15.141)

for , where is a symmetric (15.121), positive definite matrix (15.125) and is a full-rank (15.73) matrix such that , for more details see Section 15.6.2. In fact, on finite-dimensional inner product spaces, every inner product can be expressed in terms of a Mahalanobis inner product (15.141) on the vector coordinates, see Section 15.6 for details.

PICExample 15.30. Mahalanobis inner product.
We continue from Example 15.29, where we considered the vectors (15.25)-(15.26). Consider the symmetric (15.121), positive definite matrix (15.125)

(15.142)

which has inverse

(15.143)

The Mahalanobis inner product of and with scaling matrix , (15.141) is calculated as

(15.144)

Note also that while we have restricted our attention to vector spaces over the real numbers for simplicity, the concepts in this section can be easily extended to vector spaces over arbitrary fields [W], for example the field of complex numbers. This means that the scalars are elements from the chosen field, rather than the real numbers.

In an inner product space , any linear operator (15.49)-(15.50) can be expressed uniquely using the inner product . First note that the symmetry (15.135) and linearity of the inner product (15.136)-(15.137) imply that for any fixed vector , the inner product is a linear function of , i.e.

(15.145)

for . This suggests how we may use the inner product to represent a linear operator that maps to .

Indeed, for any linear operator (15.49)-(15.50) from an inner product space to the real numbers , there exists a unique vector such that E.15.11 

(15.146)

for all .

The identification between inner products and linear operators which map to the real line (15.146) generalizes to the Riesz representation theorem (17.84), which is discussed in more detail in Section 17.3.3. This result lies at the foundation of linear pricing theory (0b.25), which is covered in depth in Chapters 0b-0c.

15.3.1 Length, distance and angle

Given an inner product (15.134), we can define the associated length, which we introduce in more generality later in (15.214), or norm, of a generic vector as

(15.147)

In our simple example of a real vector space with the dot product (15.139), this length corresponds to the standard Euclidean norm

(15.148)

PICExample 15.31. Standard Euclidean norm.
Continuing from Example 15.30, the standard Euclidean norm (15.148) of the vector (15.25) is

(15.149)

Similarly, for (15.26) we obtain

(15.150)

We can similarly define the more general Mahalanobis norm on , induced by the Mahalanobis inner product (15.141) as in (15.147), giving

(15.151)

for , where is a symmetric (15.121), positive definite matrix (15.125), is the standard Euclidean norm (15.148) and a full-rank matrix such that , for more details see Section 15.6.2. On finite-dimensional inner product spaces, any norm induced by an inner product (15.147) can be expressed as the Mahalanobis norm (15.151) of the vector coordinates, see Section 15.6 for details.

PICExample 15.32. Mahalanobis norm.
Continuing from Example 15.31, the Mahalanobis norm (15.151) of the vector (15.25) with as in (15.142) is

Similarly, for (15.26) we obtain

An inner product also induces a distance, which we introduce in more generality later in (15.231), between two generic vectors via the length (15.147) of their difference, explicitly

(15.154)

for . In the example of a real vector space with the dot product, this distance (15.154) corresponds to the standard Euclidean distance

(15.155)

PICExample 15.33. Standard Euclidean distance.
Continuing from Example 15.32, we calculate the standard Euclidean distance (15.155) between the vectors (15.25)-(15.26) as

(15.156)

We can generalize the Euclidean distance (15.155) to the Mahalanobis distance on , induced by the Mahalanobis inner product (15.141) as in (15.154), giving

(15.157)

for , where is the standard Euclidean norm (15.148), is a symmetric (15.121), positive definite matrix (15.125) and is a matrix such that , for more details see Section 15.6.2. Note that the Mahalanobis distance (15.157) reduces to the standard Euclidean distance (15.155) when is the identity matrix. The Mahalanobis distance (15.157) allows us to generalize the absolute z-score (22.41) in the multivariate framework of an -dimensional random variable to the multivariate absolute z-score (22.42), as discussed in Section 22.2.

In this context the Mahalanobis distance (15.157) takes into account the correlations between the random variables, as opposed to the standard Euclidean distance (15.155) which implicitly assumes they are uncorrelated.

On finite-dimensional inner product spaces, any distance induced by an inner product (15.154) can be expressed as a Mahalanobis distance (15.157) on the coordinates, see Section 15.6 for details.

PICExample 15.34. Mahalanobis distance.
Continuing from Example 15.33, we can calculate the Mahalanobis distance (15.157) with as in (15.142) between the vectors (15.25)-(15.26) as

From the relationship between length (15.147) and inner product (15.134) we have the polarization identity [WE.15.12 

(15.159)

PICExample 15.35. Polarization identity.
Continuing from Example 15.34, we can verify the polarization identity (15.159) for the dot product (15.139) using the pair of vectors (15.25)-(15.26). Indeed, from our previous calculations of the norms (15.149) and (15.150) and the distance (15.156)

(15.160)

which agrees with our previous calculation of the inner product  (15.140).

The inner product (15.134) and length (15.147) also satisfy the Cauchy-Schwarz inequality [W]

(15.161)

where equality holds if and only if the vectors are linear dependent.

PICExample 15.36. Cauchy-Schwarz inequality.
Continuing from Example 15.35, we can verify the Cauchy-Schwarz inequality (15.161) for the dot product (15.139) using the pair of vectors (15.25)-(15.26). Indeed, from our previous calculations of the norms (15.149) and (15.150) and the inner product  (15.140), we see that

(15.162)

as the Cauchy-Schwarz inequality (15.161) states.

From the inner product (15.134) and the length (15.147) we can define the angle [W] between two generic vectors as follows

(15.163)

This is well-defined since the Cauchy-Schwartz inequality (15.161) guarantees that .

PICExample 15.37. Angle.
Continuing from Example 15.36, we can calculate the standard Euclidean angle between the pair of vectors (15.25)-(15.26). Using the previously calculated inner product  (15.140) and lengths (15.149) and (15.150), the angle between and is calculated as (15.163)

(15.164)

The notion of angle (15.163) between two vectors naturally yields the notion of orthogonality: two vectors are orthogonal with respect to the inner product (15.134) if their inner product is zero

(15.165)

Care must be taken when determining the orthogonality of vectors using their coordinates, for details see Section 15.6.1.

PICExample 15.38. Orthogonality.
Continuing from Example 15.37, we recall that the dot product of (15.25)-(15.26) is (15.140). Therefore