Linear regression is a very basic technique that we use a lot in machine learning. In a lot of cases (and I have been guilty of this), we just use it without much thought as to how the internals actually work.
In a 2-D coordinate system, we can plot observations (such as, a child’s age is 1), and associated dependent variables (ie, the child has 1 friend) on an x/y axis, like the one below:
In the above system, we have plotted several observation and dependent variable pairs [1,1], [2,4], [3,9], [4,16]. We have also added in a line. This line allows us to predict the dependent variables for future observations (when the child is 5, according to our line, they will have 20 friends). The line is defined by an equation where b is the slope of the line, and a is the y-intercept. In the simplest form of linear regression, we can figure out b and a to find the line. In more complex forms, such as , we can predict y using multiple features.
Today, I’m going to shed some light on the internals to linear regression, and do some comparisons of various approaches in Python. If you have not read my earlier post on matrix inversion, I highly recommend it.
As you can see from the line and points above, the point of linear regression is to minimize the sum of squared errors between the actual y value and the predicted y value.
We can express the various pairs in matrix form using the equation where X is the matrix of X observations, y is the vector of dependent values, b is the vector of coefficients, and e is a vector of errors (which we want to minimize).
We add in an extra column on the left of the X matrix to create the intercept, which is . We can multiply out the matrices to get:
So, we want to solve for and in a way that minimizes the error.
We can model the sum of squared errors of the above system as:
This will simplify to . Our first matrix is the transpose of the second matrix. We can denote the transpose as , and the original as E.
Error just refers to predicted values minus actual values. In this case, our actual values are in the y vector, and our predicted values are bX.
So, our squared error is .
Setting error to zero (we want to minimize it) and taking the derivative with respect to b gives us . We can then multiply this out to get , and finally .
So, this is starting to look easier. To get our coefficients, we just solve:
Implementation - Matrix class
All the implementation code is available in the algorithms repository.
In order to implement the algorithm, we first need to define a basic class for a matrix. Here is some excerpted code.
Represent a matrix and allow for basic matrix operations to be done.
def __init__(self, X):
X - a list of lists, ie [,]
#Validate that the input is ok
self.X = X
We can then write class methods to perform the operations that we need (inversion, transposition, and multiplication). Inversion was covered in a previous post, so our code is short:
Invert the matrix in place.
self.X = invert(self.X)
Transpose the matrix in place.
trans = 
for j in xrange(0,self.cols):
row = 
for i in xrange(0,self.rows):
self.X = trans
Multiplication can be added by overriding
__mul__. See the magic methods guide for information.
``` python def mul(self, Z): “”” Left hand multiplication, ie matrix * other_matrix “”” assert(isinstance(Z, Matrix))
assert Z.rows==self.cols product =  for i in xrange(0,self.rows): row =  for j in xrange(0,Z.cols): row.append(row_multiply(self.X[i], [Z[m][j] for m in xrange(0,Z.rows)])) product.append(row) return Matrix(product) ```
__rmul__ can also be overridden for right hand multiplication (and is in the algorithms repository).
Implementation – Multivariate Regression
Then, we can implement the regression.
``` python class LinregCustom(Algorithm): “”” Solves for multivariate linear regression “”” def train(self, X, y): “”” X - input list of lists y - input column vector in list form, ie [,] “”” assert len(y) == len(X) X_int = self.append_intercept(X) coefs = ((Matrix(X_int) * Matrix(X_int).transpose()).invert()) coefs = (Matrix(X_int).transpose()) * coefs coefs = coefs * Matrix(y) self.coefs = coefs
def predict(self,Z): """ Z - input list of lists """ Z = self.append_intercept(Z) return Matrix(Z) * self.coefs def append_intercept(self, X): """ Adds the intercept term to the first row of a matrix """ X = deepcopy(X) #Append this to calculate the intercept term properly for i in xrange(0,len(X)): X[i] =  + X[i] return X ```
We can try this:
lr = LinregCustom()
[[0.9999999999999991], [3.9999999999999982], [8.999999999999993], [15.999999999999996]]
Although the numbers may not be exact, you should get similar output to this. This essentially reconstructs our original inputs.
Implementation – Univariate Regression
We can fall back on a univariate linear regression formula:
``` python class LinregNonMatrix(Algorithm): “”” Solve linear regression with a single variable “”” def train(self, x, y): “”” x - a list of x values y - a list of y values “”” x_mean = mean(x) y_mean = mean(y) x_dev = sum([abs(i-x_mean) for i in x]) y_dev = sum([abs(i-y_mean) for i in y])
self.slope = (x_dev*y_dev)/(x_dev*x_dev) self.intercept = y_mean - (self.slope*x_mean) def predict(self, z): """ z - a list of x values to predict on returns - computed y values for the input vector """ return [i*self.slope + self.intercept for i in z] ```
lr = LinregNonMatrix()
[0.0, 5.0, 10.0, 15.0]
This will not pick up the clear nonlinearity in the input series, which is .
Implementation – Regression with Numpy
We can also use Numpy to solve a linear regression for us.
``` python class LinregNumpy(Algorithm): “”” Use numpy to solve a multivariate linear regression “”” def train(self,X,y): “”” X - input list of lists y - input column vector in list form, ie [,] “”” from numpy import array,linalg, ones,vstack assert len(y) == len(X) X = vstack([array(X).T,ones(len(X))]).T self.coefs = linalg.lstsq(X,y) self.coefs = self.coefs.reshape(self.coefs.shape,-1)
def predict(self,Z): """ Z - input list of lists """ from numpy import array, ones,vstack Z = vstack([array(Z).T,ones(len(Z))]).T return Z.dot(self.coefs) ```
We can reproduce the same predictions that we got with our multivariate regression.
lr = LinregNumpy()
In almost all cases, you are best off using numpy, or another linear regression algorithm that doesn’t rely on matrix inversion, and is thus faster and can solve even in situations with non-singular matrices. However, it is useful to know how linear regression is working, and it can be beneficial to tweak the formula if needed for specific situations.