The Gauss-Newton method is an iterative optimization algorithm used to solve non-linear least squares problems. It is particularly useful for curve fitting, where the goal is to find the best-fit parameters of a model function to a given set of data points. In this tutorial, we will explore the Gauss-Newton method, its mathematical derivation, and its implementation in Python with and without using optimization libraries.
Problem Statement: Let's consider a real-life problem of predicting the temperature of a city based on the day of the year. We have collected temperature data for various days throughout the year and want to fit a curve to this data to make predictions. The temperature model can be represented as a sine function:
$$ ⁍ $$
Where:
Our goal is to find the optimal values of the parameters $a$, $b$, $c$, and $e$ that minimize the sum of squared residuals between the observed temperatures and the predicted temperatures.
Mathematical Derivation: The Gauss-Newton method is based on the idea of linearizing the non-linear model function around the current estimate of parameters and iteratively updating the parameters to minimize the sum of squared residuals.
Let's denote the residual function as:
$$ ⁍ $$
where $T_i$ is the observed temperature on day $d_i$, $T(d_i, \mathbf{p})$ is the predicted temperature using the current parameter vector $\mathbf{p} = [a, b, c, e]^T$, and $r_i$ is the residual.
The sum of squared residuals is given by:
$$ ⁍ $$
To minimize $S(\mathbf{p})$, the Gauss-Newton method linearizes the model function using a first-order Taylor expansion around the current parameter estimate $\mathbf{p}_k$:
$$ ⁍ $$
where $\mathbf{J}_i(\mathbf{p}_k)$ is the Jacobian matrix evaluated at $\mathbf{p}_k$.