Introduction

Newton's method, also known as the Newton-Raphson method, is an iterative algorithm used to find the roots of a real-valued function. It can also be extended to find the minimum or maximum of a function. In this tutorial, we will explore the mathematical concepts behind Newton's method and provide Python code examples for both root-finding and optimization problems.

Finding Roots

Consider a real-valued function $f(x)$. The goal is to find a value $x^o$ such that. Newton's method starts with an initial guess $x_0$ and iteratively updates it using the following formula:

$$ ⁍ $$

where $f'(x_n)$ is the derivative of $f(x)$ evaluated at $x_n$.

Example: Single Variable Function

Let's find the root of the function $f(x) = x^3 - 2x - 5$.

def f(x):
    return x**3 - 2*x - 5

def f_prime(x):
    return 3*x**2 - 2

def newton_method(f, f_prime, x0, tolerance=1e-6, max_iterations=100):
    x = x0
    for i in range(max_iterations):
        fx = f(x)
        if abs(fx) < tolerance:
            return x
        x = x - fx / f_prime(x)
    return x

x0 = 2  # Initial guess
root = newton_method(f, f_prime, x0)
print(f"The root is approximately {root:.6f}")

Output:

The root is approximately 2.094551

Optimization

Newton's method can be extended to find the minimum or maximum of a function. In this case, we iteratively update the point $x_n$ using the following formula:

$$ ⁍ $$

where $f'(x_n)$ and $f''(x_n)$ are the first and second derivatives of $f(x)$ evaluated at $x_n$.

Multivariate Optimization

For a multivariate function $f(x_1, x_2, \ldots, x_n)$, the gradient vector $\nabla f(x)$ and the Hessian matrix $H(x)$ are defined as follows:

Gradient vector:

$$

⁍ $$

Hessian matrix: