Introduction

Linear regression is a fundamental statistical method used to model the relationship between a dependent variable and one or more independent variables. It is widely used for prediction, trend analysis, and inferential statistics. This tutorial will cover the theory behind linear regression, the least squares method, assumptions, evaluation metrics, and practical implementation in Python with detailed examples.

Table of Contents

1. What is Linear Regression?
2. The Least Squares Method
3. Assumptions of Linear Regression
4. Simple Linear Regression
5. Multiple Linear Regression
6. Evaluating Model Performance
7. Fitting a Straight Line to Points
8. Applications of Linear Regression
9. Python Implementation

1. What is Linear Regression?

Linear regression aims to find the linear relationship that best fits the data. It involves modeling the relationship between a dependent variable $Y$ and one or more independent variables $X$.

Linear regression can be categorized into:

• Simple Linear Regression: Modeling the relationship between a single independent variable and a dependent variable.
• Multiple Linear Regression: Modeling the relationship between multiple independent variables and a dependent variable.

2. The Least Squares Method

The least squares method minimizes the sum of the squared differences between observed and predicted values. This is achieved by finding the line (or hyperplane in multiple dimensions) that minimizes the sum of squared residuals.

Mathematically, the least squares problem can be expressed as: $\min_{\theta} \| Y - H\theta \|^2$ where: