Theory of functional connections

The Theory of Functional Connections (TFC) is a mathematical framework designed for functional interpolation. It introduces a method to derive a functional— a function that operates on another function—capable of transforming constrained optimization problems into equivalent unconstrained problems. This transformation enables the application of TFC to various mathematical challenges, including the solution of differential equations. Functional interpolation, in this context, refers to constructing functionals that always satisfy given constraints, regardless of the expression of the internal (free) function.

From interpolation to functional interpolation

To provide a general context for the TFC, consider a generic interpolation problem involving $n$ constraints, such as a differential equation subject to a boundary value problem (BVP). Regardless of the differential equation, these constraints may be consistent or inconsistent. For instance, in a problem over the domain ${\mathcal {D}}:(0,1)\cup (0,1)$, the constraints $f_{1}(x,0)=1+x$ and $f_{2}(0,y)=2-y$ are inconsistent, as they yield different values at the shared point $(0,0)$. If the $n$ constraints are consistent, a function interpolating these constraints can be constructed by selecting $n$ linearly independent support functions, such as monomials, $\{1,x,x^{2},\cdots ,x^{n-1}\}$. The chosen set of support functions may or may not be consistent with the given constraints. The consistency problem is addressed by examining constraints, interpolation, and functional interpolation cases, including scenarios where boundary conditions involve shear and mixed derivatives. For instance, the constraints $y(-1)=y(+1)=0$ and ${\dfrac {dy}{dx}}{\bigg |}_{x=0}=1$ are inconsistent with the support functions, $\{1,x,x^{2}\}$, as can be easily verified. If the support functions are consistent with the constraints, the interpolation problem can be solved, yielding an interpolant--a function that satisfies all constraints. Choosing a different set of support functions would result in a different interpolant. When an interpolation problem is solved and an initial interpolant is determined, all possible interpolants can, in principle, be generated by performing the interpolation process with every distinct set of linearly independent support functions consistent with the constraints. However, this method is impractical, as the number of possible sets of support functions is infinite.

This challenge was addressed through the development of the TFC, an analytical framework for performing functional interpolation introduced by Daniele Mortari at Texas A&M University. The approach involves constructing a functional $f{\big (}\mathbf {x} ,g(\mathbf {x} ){\big )}$ that satisfies the given constraints for any arbitrary expression of $g(\mathbf {x} )$, referred to as the free function. This functional, known as the constrained functional, provides a complete representation of all possible interpolants. By varying $g(\mathbf {x} )$, it is possible to generate the entire set of interpolants, including those that are discontinuous or partially defined.

Function and functional interpolation flowchart

Function interpolation produces a single interpolating function, while functional interpolation generates a family of interpolating functions represented through a functional. This functional defines the subspace of functions that inherently satisfy the given constraints, effectively reducing the solution space to the region where solutions to the constrained optimization problem are located. By employing these functionals, constrained optimization problems can be reformulated as unconstrained problems. This reformulation allows for simpler and more efficient solution methods, often improving accuracy, robustness, and reliability. Within this context, the Theory of Functional Connections (TFC) provides a systematic framework for transforming constrained problems into unconstrained ones, thereby streamlining the solution process.

TFC addresses univariate constraints involving points, derivatives, integrals, and any linear combination of these. The theory is also extended to accommodate infinite and multivariate constraints and applied to solving ordinary, partial, and integro-differential equations. The univariate version of TFC can be expressed in one of the following two forms:

${\begin{cases}f{\big (}x,g(x){\big )}=g(x)+\displaystyle \sum _{j=1}^{n}\eta _{j}{\big (}x,g(x){\big )}\,s_{j}(x)\\f{\big (}x,g(x){\big )}=g(x)+\displaystyle \sum _{j=1}^{n}\phi _{j}{\big (}x,\mathbf {s} (x){\big )}\,\rho _{j}{\big (}x,g(x){\big )},\end{cases}}$

where $n$ represents the number of linear constraints, $g(x)$ is the free function, and $s_{j}(x)$ are $n$ user-defined, linearly independent support functions. The terms $\eta _{j}(x,g(x))$ are the coefficient functionals, $\phi _{j}(x)$ are switching functions (which take a value of 1 when evaluated at their respective constraint and 0 at other constraints), and $\rho _{j}{\big (}x,g(x){\big )}$ are projection functionals that express the constraints in terms of the free function.

A rational example

To show how TFC generalizes interpolation, consider the constraints, ${\dot {y}}(x_{1})={\dot {y}}_{1}$ and ${\dot {y}}(x_{2})={\dot {y}}_{2}$. An interpolating function satisfying these constraints is,

$f_{a}(x)={\dfrac {x(2x_{2}-x)}{2(x_{2}-x_{1})}}\,{\dot {y}}_{1}+{\dfrac {x(x-2x_{1})}{2(x_{2}-x_{1})}}\,{\dot {y}}_{2},$

as can be easily verified. Because of this interpolation property, the derivative of the function,

$\delta (x)=g(x)-{\dfrac {x(2x_{2}-x)}{2(x_{2}-x_{1})}}\,{\dot {g}}(x_{1})-{\dfrac {x(x-2x_{1})}{2(x_{2}-x_{1})}}\,{\dot {g}}(x_{2}),$

vanishes at $x_{1}$ and $x_{2}$, for \textit{any} function, $g(x)$. Therefore, by adding $\delta (x)$ to $f_{a}(x)$, a functional is obtained that still satisfies the constraints,

$f{\big (}x,g(x){\big )}=f_{a}(x)+\delta (x)={\dfrac {x(2x_{2}-x)}{2(x_{2}-x_{1})}}\,{\dot {y}}_{1}+{\dfrac {x(x-2x_{1})}{2(x_{2}-x_{1})}}\,{\dot {y}}_{2}+g(x)-{\dfrac {x(2x_{2}-x)}{2(x_{2}-x_{1})}}\,{\dot {g}}(x_{1})-{\dfrac {x(x-2x_{1})}{2(x_{2}-x_{1})}}\,{\dot {g}}(x_{2}),$

no matter what $g(x)$ is. Due to this property, this functional is referred to as constrained functional. The key requirement for the functional $f{\big (}x,g(x){\big )}$ to work as intended is that the terms ${\dot {g}}(x_{1})$ and ${\dot {g}}(x_{2})$ are defined. Once this condition is met, the functional $f{\big (}x,g(x){\big )}$ is free to take on any arbitrary values beyond the specified constraints, thanks to the infinite flexibility provided by $g(x)$. Importantly, this flexibility is not limited to the specific constraints chosen in this example. Instead, it applies universally to any set of constraints. This universality illustrates how TFC performs functional interpolation: it constructs a function that satisfies the given constraints while simultaneously allowing complete freedom in behavior elsewhere through the choice of $g(x)$. In essence, this example demonstrates that the constrained functional $f{\big (}x,g(x){\big )}$ captures all possible functions that meet the given constraints, showcasing the power and generality of TFC in handling a wide variety of interpolation problems.

Example: A univariate constrained functional animation using 2 absolute constraints and one relative constraint.

Applications of TFC

TFC has been extended and employed in various applications, including its use in shear-type and mixed derivative problems, the analysis of fractional operators, the determination of geodesics for BVP in curved spaces, and the contribution to continuation methods. Additionally, TFC has been applied to indirect optimal control, the modeling of stiff chemical kinetics, and the study of epidemiological dynamics. It has also demonstrated potential in nonlinear programming and structural mechanics and radiative transfer, among other areas. An efficient, free TFC toolbox is available at https://github.com/leakec/tfc.

Of particular note is the application of TFC in neural networks, where it has shown exceptional efficiency, especially addressing high-dimensional problems and in enhancing the performance of physics-informed neural networks by effectively eliminating constraints from the optimization process, a challenge that traditional neural networks often struggle to address. This capability significantly improves computational efficiency and accuracy, enabling the resolution of complex problems with greater ease. TFC has been employed with physics-informed neural networks and symbolic regression techniques for physics discovery of dynamical systems.

Difference with spectral methods

At first glance, TFC and spectral methods may appear similar in their approach to solving constrained optimization problems. However, there are two fundamental distinctions between them:

• Representation of solutions: Spectral methods represent the solution as a sum of basis functions, whereas TFC represents the free function as a sum of basis functions. This distinction allows TFC to analytically satisfy the constraints, while spectral methods treat constraints as additional data, approximating them with an accuracy dependent on the residuals.
• Computational approach in BVP: In linear BVPs, the computational strategies of the two methods differ significantly. Spectral methods typically employ iterative techniques, such as the shooting method, to reformulate the BVP as an initial value problem, which is simpler to solve. Conversely, TFC directly addresses these problems through linear least-squares techniques, avoiding the need for iterative procedures.

Both methods can perform optimization using either the Galerkin method, which ensures the residual vector is orthogonal to the chosen basis functions, or the Collocation method, which minimizes the norm of the residual vector.

Difference with Lagrange multipliers technique

The Lagrange multipliers method is a widely used approach for imposing constraints in an optimization problem. This technique introduces additional variables, known as multipliers, which must be computed to enforce the constraints. While the computation of these multipliers is straightforward in some cases, it can be challenging or even practically infeasible in others, thereby adding significant complexity to the problem. In contrast, TFC doesn't add new variables and enables the derivation of constrained functionals without encountering insurmountable difficulties. However, it is important to note that the Lagrange multiplier method has the advantage of handling inequality constraints, a capability that TFC currently lacks.

A notable limitation of both approaches is their propensity to produce solutions that correspond to local optima rather than guaranteed global optima, particularly in the context of non-convex problems. Consequently, supplementary verification procedures or alternative methods may be required to assess and confirm the quality and global validity of the obtained solution. In summary, while TFC does not entirely replace the Lagrange multipliers method, it serves as a powerful alternative in cases where the computation of multipliers becomes excessively complex or infeasible, provided the constraints are limited to equalities.