Quora noscript

29.8 Functional calculusPIC

Key points

  • The directional derivative (29.186) becomes the Gateaux derivative (29.188) in an function space.
  • The gradient (29.190) becomes the Frechet derivative (29.196) in an function space, and the Hessian (29.202) becomes the second-order Frechet derivative (29.206).

In multivariate calculus, discussed in Section 28.1, we start from a function  (28.22) that takes as input a vector with entries

(29.180)

Consider a perturbation vector of size defined by the standard Euclidean inner product (27.128)

(29.181)

with small. The purpose of calculus is to study the effect of the perturbation (29.181) on the function (29.180) when applied to a generic vector

(29.182)

Here we generalize calculus, which operates on , to functional analysis, which operates on a space of functions, such as the square-integrable functions (29.48), as in Table 29.1.

To this purpose, let us use the notation for vectors in , rather than , and let us interpret a vector as a function (29.26). Then, the notion of multivariate function (29.180) generalizes to that of functional, which maps a function to a complex number

(29.183)

Consider a perturbation vector of size defined by the inner product (29.43)

(29.184)

with small. The purpose of functional calculus is to study the effect of the perturbation (29.184) on the functional (29.183) when applied to a generic function

(29.185)

29.8.1 Gateaux derivative

The Gateaux derivative is the functional analysis generalization of the directional derivative.

In multivariate calculus we compute the directional derivative (28.42) of a function (29.180) in the direction of the vector as follows

(29.186)

Then the partial derivative (28.23) is the special case of a directional derivative (28.43) in the direction of the canonical basis vector (27.28)

(29.187)

The Gateaux derivative [W] of a functional (29.183) in the direction of the function (29.48) is defined similarly, as follows

(29.188)

In particular, we denote the Gateaux derivative in the direction of the Dirac delta (29.53) as follows

(29.189)

29.8.2 Frechet derivative

The Frechet derivative is the functional analysis generalization of the gradient.

In multivariate calculus we can stack the partial derivatives (29.187) in the gradient (28.29)

(29.190)

or equivalently

(29.191)

for all . Hence, we can interpret the gradient (29.190) as a map from the vector space into itself

(29.192)

The gradient (29.190) recovers the directional derivative (29.186) as in (28.44)

(29.193)

where is the standard Euclidean dot product (27.128)

(29.194)

With the gradient (29.190) we can perform a first order Taylor expansion (28.136) of a function (29.180), which we rewrite in compact form as

(29.195)

where is the norm induced by the dot product (29.194), and denotes terms smaller than [W].

In the context of functional analysis, the Frechet derivative [W] is a map from the function space (29.48) into itself

(29.196)

that collects all the Gateaux derivatives in the direction of the Dirac delta (29.189)

(29.197)

for all .

Note that, unlike in the finite dimensional case, there may be occurrences where the Gateaux derivatives are well defined and yet the Frechet derivative (29.197) is not well defined [W], though such occurrences are beyond the scope of the present discussion. When the Frechet derivative (29.196) is well defined, it recovers the Gateaux derivative in a generic direction (29.188)

(29.198)

where is the inner product (29.43)

(29.199)

With the Frechet derivative (29.196) we can perform a first order Taylor expansion of a functional (29.195)

(29.200)

where is the norm induced by the inner product (29.199).

29.8.3 Second order derivative

The second order Frechet derivative is the functional analysis generalization of the Hessian.

In multivariate calculus, it is convenient to arrange the second-order derivatives in matrix form in the Hessian (28.50). We can interpret the Hessian as a map from the vector space into a matrix, which means a linear application from the vector space in itself

(29.201)

where . The Hessian (29.201) operates on an arbitrary vector via the matrix-vector multiplication (27.62)

(29.202)

With the Hessian (29.201) we can perform a second-order Taylor expansion (28.137) of a function (29.180), which we rewrite in compact form as

(29.203)

The left hand side in (29.203) are the terms of the first order Taylor expansion (29.200). The calculation of the inner product on the right hand side follows from the linear action (29.202) and the dot product (27.128)

(29.204)

In the context of functional analysis, the second order Frechet derivative [ W] is a map from the function space (29.48) into a kernel, which means a linear operator from the function space into itself

(29.205)

The second order Frechet derivative (29.205) operates on an arbitrary vector via kernel-function integration (29.39)

(29.206)

With the second order Frechet derivative (29.205) we can perform a second-order Taylor expansion of a functional (29.203)

(29.207)

The left hand side in (29.207) are the terms of the first order Taylor expansion (29.200). The calculation of the inner product on the right hand side follows from the linear action (29.206) and the inner product (29.43)

(29.208)