Back to Study Material Machine Learning

Simple linear regression

Simple linear regression is a supervised learning algorithm that models the relationship between one input feature and one continuous output by fitting a s...

Simple linear regression

Fitting a Straight Line Through Data

Simple linear regression is a supervised learning algorithm that models the relationship between one input feature and one continuous output by fitting a straight line to the training data. Despite its simplicity, it remains one of the most widely used statistical tools in science, economics, and engineering because it is transparent, computationally cheap, and produces directly interpretable results.

The model has the form y equals w times x plus b, where x is the input feature, y is the predicted output, w is the slope or weight, and b is the intercept or bias. Training the model means finding the values of w and b that minimize the difference between the predicted values and the true labels across all training examples.

The most common way to measure that difference is the sum of squared errors. Squaring the individual errors ensures that positive and negative deviations do not cancel each other out and penalizes large errors more heavily than small ones. The values of w and b that minimize this sum can be found analytically using the ordinary least squares formula, without iterating through gradient descent.

Interpretation is straightforward. The slope w tells you how much the output is expected to change for a one-unit increase in the input. If you are modeling house price against square footage and the slope is 150, each additional square foot is associated with a 150-unit increase in predicted price.

A concrete example: a study of student exam scores versus hours of study fits a line with slope 8 and intercept 20. A student who studies 5 hours is predicted to score 60. The model is easy to explain to a non-technical audience and quick to validate against domain knowledge.

Exam tip: know that ordinary least squares minimizes the sum of squared vertical residuals, not horizontal distances, and that the assumptions of simple linear regression include linearity, constant variance of errors, and independence of observations.