Functional regression

Functional linear models with scalar responses can be obtained by replacing the scalar covariates ${\displaystyle X}$ and the coefficient vector ${\displaystyle \beta }$ in model (1) by a centered functional covariate ${\displaystyle X^{c}(\cdot )=X(\cdot )-\mathbb {E} (X(\cdot ))}$ and a coefficient function ${\displaystyle \beta =\beta (\cdot )}$ with domain ${\displaystyle {\mathcal {T}}}$, respectively, and replacing the inner product in Euclidean space by that in Hilbert space ${\displaystyle L^{2}}$,

$${\displaystyle Y=\beta _{0}+\langle X^{c},\beta \rangle +\varepsilon =\beta _{0}+\int _{\mathcal {T}}X^{c}(t)\beta (t)\,dt+\varepsilon ,}$$

(2)

where ${\displaystyle \langle \cdot ,\cdot \rangle }$ here denotes the inner product in ${\displaystyle L^{2}}$. One approach to estimating ${\displaystyle \beta _{0}}$ and ${\displaystyle \beta (\cdot )}$ is to expand the centered covariate ${\displaystyle X^{c}(\cdot )}$ and the coefficient function ${\displaystyle \beta (\cdot )}$ in the same functional basis, for example, B-spline basis or the eigenbasis used in the Karhunen–Loève expansion. Suppose ${\displaystyle \{\phi _{k}\}_{k=1}^{\infty }}$ is an orthonormal basis of ${\displaystyle L^{2}}$. Expanding ${\displaystyle X^{c}}$ and ${\displaystyle \beta }$ in this basis, ${\displaystyle X^{c}(\cdot )=\sum _{k=1}^{\infty }x_{k}\phi _{k}(\cdot )}$, ${\displaystyle \beta (\cdot )=\sum _{k=1}^{\infty }\beta _{k}\phi _{k}(\cdot )}$, model (2) becomes

$${\displaystyle Y=\beta _{0}+\sum _{k=1}^{\infty }\beta _{k}x_{k}+\varepsilon .}$$

For implementation, regularization is needed and can be done through truncation, ${\displaystyle L^{2}}$ penalization or ${\displaystyle L^{1}}$ penalization.[1] In addition, a reproducing kernel Hilbert space (RKHS) approach can also be used to estimate ${\displaystyle \beta _{0}}$ and ${\displaystyle \beta (\cdot )}$ in model (2)[2]

Adding multiple functional and scalar covariates, model (2) can be extended to

$${\displaystyle Y=\sum _{k=1}^{q}Z_{k}\alpha _{k}+\sum _{j=1}^{p}\int _{{\mathcal {T}}_{j}}X_{j}^{c}(t)\beta _{j}(t)\,dt+\varepsilon ,}$$

(3)

where ${\displaystyle Z_{1},\ldots ,Z_{q}}$ are scalar covariates with ${\displaystyle Z_{1}=1}$, ${\displaystyle \alpha _{1},\ldots ,\alpha _{q}}$ are regression coefficients for ${\displaystyle Z_{1},\ldots ,Z_{q}}$, respectively, ${\displaystyle X_{j}^{c}}$ is a centered functional covariate given by ${\displaystyle X_{j}^{c}(\cdot )=X_{j}(\cdot )-\mathbb {E} (X_{j}(\cdot ))}$, ${\displaystyle \beta _{j}}$ is regression coefficient function for ${\displaystyle X_{j}^{c}(\cdot )}$, and ${\displaystyle {\mathcal {T}}_{j}}$ is the domain of ${\displaystyle X_{j}}$ and ${\displaystyle \beta _{j}}$, for ${\displaystyle j=1,\ldots ,p}$. However, due to the parametric component ${\displaystyle \alpha }$, the estimation methods for model (2) cannot be used in this case[3] and alternative estimation methods for model (3) are available.[4][5]

For a functional response ${\displaystyle Y(\cdot )}$ with domain ${\displaystyle {\mathcal {T}}}$ and a functional covariate ${\displaystyle X(\cdot )}$ with domain ${\displaystyle {\mathcal {S}}}$, two FLMs regressing ${\displaystyle Y(\cdot )}$ on ${\displaystyle X(\cdot )}$ have been considered.[3][6] One of these two models is of the form

$${\displaystyle Y(t)=\beta _{0}(t)+\int _{\mathcal {S}}\beta (s,t)X^{c}(s)\,ds+\varepsilon (t),\ {\text{for}}\ t\in {\mathcal {T}},}$$

(4)

where ${\displaystyle X^{c}(\cdot )=X(\cdot )-\mathbb {E} (X(\cdot ))}$ is still the centered functional covariate, ${\displaystyle \beta _{0}(\cdot )}$ and ${\displaystyle \beta (\cdot ,\cdot )}$ are coefficient functions, and ${\displaystyle \varepsilon (\cdot )}$ is usually assumed to be a random process with mean zero and finite variance. In this case, at any given time ${\displaystyle t\in {\mathcal {T}}}$, the value of ${\displaystyle Y}$, i.e., ${\displaystyle Y(t)}$, depends on the entire trajectory of ${\displaystyle X}$. Model (4), for any given time ${\displaystyle t}$, is an extension of multivariate linear regression with the inner product in Euclidean space replaced by that in ${\displaystyle L^{2}}$. An estimating equation motivated by multivariate linear regression is

$${\displaystyle r_{XY}=R_{XX}\beta ,{\text{ for }}\beta \in L^{2}({\mathcal {S}}\times {\mathcal {S}}),}$$

where ${\displaystyle r_{XY}(s,t)={\text{cov}}(X(s),Y(t))}$, ${\displaystyle R_{XX}:L^{2}({\mathcal {S}}\times {\mathcal {S}})\rightarrow L^{2}({\mathcal {S}}\times {\mathcal {T}})}$ is defined as ${\displaystyle (R_{XX}\beta )(s,t)=\int _{\mathcal {S}}r_{XX}(s,w)\beta (w,t)dw}$ with ${\displaystyle r_{XX}(s,w)={\text{cov}}(X(s),X(w))}$ for ${\displaystyle s,w\in {\mathcal {S}}}$.[3] Regularization is needed and can be done through truncation, ${\displaystyle L^{2}}$ penalization or ${\displaystyle L^{1}}$ penalization.[1] Various estimation methods for model (4) are available.[7][8]
When ${\displaystyle X}$ and ${\displaystyle Y}$ are concurrently observed, i.e., ${\displaystyle {\mathcal {S}}={\mathcal {T}}}$,[9] it is reasonable to consider a historical functional linear model, where the current value of ${\displaystyle Y}$ only depends on the history of ${\displaystyle X}$, i.e., ${\displaystyle \beta (s,t)=0}$ for ${\displaystyle s>t}$ in model (4).[3][10] A simpler version of the historical functional linear model is the functional concurrent model (see below).
Adding multiple functional covariates, model (4) can be extended to

$${\displaystyle Y(t)=\beta _{0}(t)+\sum _{j=1}^{p}\int _{{\mathcal {S}}_{j}}\beta _{j}(s,t)X_{j}^{c}(s)\,ds+\varepsilon (t),\ {\text{for}}\ t\in {\mathcal {T}},}$$

(5)

where for ${\displaystyle j=1,\ldots ,p}$, ${\displaystyle X_{j}^{c}(\cdot )=X_{j}(\cdot )-\mathbb {E} (X_{j}(\cdot ))}$ is a centered functional covariate with domain ${\displaystyle {\mathcal {S}}_{j}}$, and ${\displaystyle \beta _{j}(\cdot ,\cdot )}$ is the corresponding coefficient function with the same domain, respectively.[3] In particular, taking ${\displaystyle X_{j}(\cdot )}$ as a constant function yields a special case of model (5)

$${\displaystyle Y(t)=\sum _{j=1}^{p}X_{j}\beta _{j}(t)+\varepsilon (t),\ {\text{for}}\ t\in {\mathcal {T}},}$$

which is a FLM with functional responses and scalar covariates.

Assuming that ${\displaystyle {\mathcal {S}}={\mathcal {T}}}$, another model, known as the functional concurrent model, sometimes also referred to as the varying-coefficient model, is of the form

$${\displaystyle Y(t)=\alpha _{0}(t)+\alpha (t)X(t)+\varepsilon (t),\ {\text{for}}\ t\in {\mathcal {T}},}$$

(6)

where ${\displaystyle \alpha _{0}}$ and ${\displaystyle \alpha }$ are coefficient functions. Note that model (6) assumes the value of ${\displaystyle Y}$ at time ${\displaystyle t}$, i.e., ${\displaystyle Y(t)}$, only depends on that of ${\displaystyle X}$ at the same time, i.e., ${\displaystyle X(t)}$. Various estimation methods can be applied to model (6).[11][12][13]
Adding multiple functional covariates, model (6) can also be extended to

$${\displaystyle Y(t)=\alpha _{0}(t)+\sum _{j=1}^{p}\alpha _{j}(t)X_{j}(t)+\varepsilon (t),\ {\text{for}}\ t\in {\mathcal {T}},}$$

where ${\displaystyle X_{1},\ldots ,X_{p}}$ are multiple functional covariates with domain ${\displaystyle {\mathcal {T}}}$ and ${\displaystyle \alpha _{0},\alpha _{1},\ldots ,\alpha _{p}}$ are the coefficient functions with the same domain.[3]

Functional polynomial models are an extension of the FLMs with scalar responses, analogous to extending linear regression to polynomial regression. For a scalar response ${\displaystyle Y}$ and a functional covariate ${\displaystyle X(\cdot )}$ with domain ${\displaystyle {\mathcal {T}}}$, the simplest example of functional polynomial models is functional quadratic regression[14]

$${\displaystyle Y=\alpha +\int _{\mathcal {T}}\beta (t)X^{c}(t)\,dt+\int _{\mathcal {T}}\int _{\mathcal {T}}\gamma (s,t)X^{c}(s)X^{c}(t)\,ds\,dt+\varepsilon ,}$$

where ${\displaystyle X^{c}(\cdot )=X(\cdot )-\mathbb {E} (X(\cdot ))}$ is the centered functional covariate, ${\displaystyle \alpha }$ is a scalar coefficient, ${\displaystyle \beta (\cdot )}$ and ${\displaystyle \gamma (\cdot ,\cdot )}$ are coefficient functions with domains ${\displaystyle {\mathcal {T}}}$ and ${\displaystyle {\mathcal {T}}\times {\mathcal {T}}}$, respectively, and ${\displaystyle \varepsilon }$ is a random error with mean zero and finite variance. By analogy to FLMs with scalar responses, estimation of functional polynomial models can be obtained through expanding both the centered covariate ${\displaystyle X^{c}}$ and the coefficient functions ${\displaystyle \beta }$ and ${\displaystyle \gamma }$ in an orthonormal basis.[14]

A functional multiple index model is given by

$${\displaystyle Y=g\left(\int _{\mathcal {T}}X^{c}(t)\beta _{1}(t)\,dt,\ldots ,\int _{\mathcal {T}}X^{c}(t)\beta _{p}(t)\,dt\right)+\varepsilon .}$$

Taking ${\displaystyle p=1}$ yields a functional single index model. However, for ${\displaystyle p>1}$, this model is problematic due to curse of dimensionality. With ${\displaystyle p>1}$ and relatively small sample sizes, the estimator given by this model often has large variance.[15] An alternative ${\displaystyle p}$-component functional multiple index model can be expressed as

$${\displaystyle Y=g_{1}\left(\int _{\mathcal {T}}X^{c}(t)\beta _{1}(t)\,dt\right)+\cdots +g_{p}\left(\int _{\mathcal {T}}X^{c}(t)\beta _{p}(t)\,dt\right)+\varepsilon .}$$

Estimation methods for functional single and multiple index models are available.[15][16]

Given an expansion of a functional covariate ${\displaystyle X}$ with domain ${\displaystyle {\mathcal {T}}}$ in an orthonormal basis ${\displaystyle \{\phi _{k}\}_{k=1}^{\infty }}$: ${\displaystyle X(t)=\sum _{k=1}^{\infty }x_{k}\phi _{k}(t)}$, a functional linear model with scalar responses shown in model (2) can be written as

$${\displaystyle \mathbb {E} (Y|X)=\mathbb {E} (Y)+\sum _{k=1}^{\infty }\beta _{k}x_{k}.}$$

One form of FAMs is obtained by replacing the linear function of ${\displaystyle x_{k}}$, i.e., ${\displaystyle \beta _{k}x_{k}}$, by a general smooth function ${\displaystyle f_{k}}$,

$${\displaystyle \mathbb {E} (Y|X)=\mathbb {E} (Y)+\sum _{k=1}^{\infty }f_{k}(x_{k}),}$$

where ${\displaystyle f_{k}}$ satisfies ${\displaystyle \mathbb {E} (f_{k}(x_{k}))=0}$ for ${\displaystyle k\in \mathbb {N} }$.[3][17] Another form of FAMs consists of a sequence of time-additive models:

$${\displaystyle \mathbb {E} (Y|X(t_{1}),\ldots ,X(t_{p}))=\sum _{j=1}^{p}f_{j}(X(t_{j})),}$$

where ${\displaystyle \{t_{1},\ldots ,t_{p}\}}$ is a dense grid on ${\displaystyle {\mathcal {T}}}$ with increasing size ${\displaystyle p\in \mathbb {N} }$, and ${\displaystyle f_{j}(x)=g(t_{j},x)}$ with ${\displaystyle g}$ a smooth function, for ${\displaystyle j=1,\ldots ,p}$[3][18]