What is regression? A very common statistics question is whether or not one variable is associated with another variable, and if it is, can we use it to predict the other? Is the temperature outside associated with ice cream sales? Regression analysis is the statistical tool that allows us to quantify the relationship between a particular variable

Linear regression is a technique that estimates the liner relationship between a continuous dependent variable (think of Y) and one independent variable (X). These independent variables are often represented by the letter X. The dependent variable is also known as the outcome variable, and is commonly represented by the letter Y. So, for example, as X increases, Y increases. As the outside temperature increases, ice cream sales increase. As a person’s amount of exercise increases, the chances of heart disease decreases.

However, simple linear regression has only one independent variable, and one continuous dependent variable. With regression, we are trying to predict the Y variable from X using a linear relationship in the form of a line, as below.

The term b not (b subscript 0) is a constant that’s known as the intercept and the b one (b subscript 1) is the slope. Y is known as the response or dependent variable and X is known as the predictor or independent variable. The machine learning community tends to use other terms, calling the Y the target and the X the feature vector.

Simple linear regression aims to find the best line to predict Y. The predictions are fitted values and the prediction errors are residuals. A common technique for calculating the coefficients of a linear regression model is called ordinary least squares, (OLS). The regression line is the estimate that minimizes the sum of the squared residual values. It’s also called the residual sum of squares or RSS.

In simple linear regression, we are only interested in two variables, one X and one Y.