Learning to implement Linear Regression from scratch in python.
In this post, I will explain what linear regression is and the basic assumptions that a linear regression model makes, and also I will build a linear regression model from scratch.
Linear regression is basically the easiest and the most intuitive machine learning model, It’s basically the hello world of machine learning. Probably everyone started with machine learning from linear regression. But we are not going to learn It like others we are going to understand the mechanics of how linear regression works under the hood then we are going to implement it in python from scratch, we are still going to use numpy library for better performance.
Starting from the basics how does Linear Regression algorithm actually work? The idea is very simple Linear regression is nothing but a function which takes data X of certain shape (n,m) where n is the number of rows and m is the number of columns, after some calculation it will return a value Y. The input can contain multiple rows which indicate multiple instances or records and the number of values returned in Y is equal to the number of records passed.
So Linear regression has two major components B0 and B1, First understand that the entire linear regression model is a best fit line, B0 is called a constant or an intercept which indicates the location of the line in the x-axis where it intersects. B1 is also called the slope of the line it indicates the steepness of the line, we will discuss the significance of the slope when we will learn about gradient descent. For now let’s try to implement this equation as a function in python.
What we are doing here is we are iterating through every column in X and assigning the columns in this equation and calculating the yhat which are the target values that are predicted by the model.
Now that we have the predicted values let’s see how far are the predicted values from the actual values, one way to do this is to calculate mean squared error. This will quantify the distance between the predicted values and target values. Let’s try to implement that in python.
Now we have our mean squared error score, we will use this to improve our model. One thing to note is that the performance of the entire model depends on 2 parameters B1 and B0, in order to improve the performance we have to update these based on error score. Now we are going to use an algorithm called gradient descent.
Gradient descent is a very simple algorithm, all it does is it tries to minimize a function. In our problem we can use gradient descent to minimize the error which we just calculated above.
In the above formula, first we are calculating the derivative of the parameter with respect to the error and then we are multiplying it with a number called learning rate, we use it to control the movement or jump of the gradient descent. And lastly we add it with the current parameter and re-assign it.
We repeat the above steps until we reach the global minimum i.e. error so small that we cannot make it smaller any further. Once we have reached that point we can stop the algorithm and use the current parameters for further predictions. Now let’s implement this in python.
After around 100 or 1000 iterations we can minimize the error and use the current parameters to predict further.
This is the entire code for linear regression.
Conclusion:
So this is how Linear regression works, This is a great machine learning model to start with in most regression problems. It’s simplicity allows us to deeply understand how the model work from inside and interpret easily. You can modify the code and add some more evaluation metrics like root mean squared error and mean absolute error.
