Lately I've been diving into the CS229 - The famous Machine Learning course taught by Andrew Ng.
While the deep theory and complex math makes me struggle, I still wanted to bring it to life by actually coding the algorithms from scratch.
In this post, I'll walk you through building a Linear Regression Model from scratch!
No Libraries, No Frameworks - Just pure math and Python.
Step 1: Defining the Training Data
X = [[1], [2], [3], [4], [5]]
y = [2, 4, 6, 8, 10]
Here X denotes the inputs and y is the corresponding output.
Our goal is to learn a mapping from X to y.
Step 2: Bias Term and Theta
A bias term is now added to make our model learn where the line should start (not just its slope).
This is done by adding 1 to every input term.
Before: X = [ [1], [2], [3] ]
After: X= [ [1,1], [1,2], [1,3] ]
Let's also define theta as [0.0,0.0] since they represent the parameters of our model i.e. the intercept and the slope respectively.
Step 3: Defining the Cost Function
Let's define the hyperparameters and a cost function to measure our model's error.
A cost function measures how wrong a model’s predictions are compared to actual results i.e. it quantifies the error to help the model improve.
The goal is to minimize the cost function by adjusting(modifying) the parameters of the model.
Step 4: Implementing Gradient Descent Algorithm
Now I'll be implementing a gradient descent algorithm - iteratively adjusting model parameters to reduce the cost function J(θ). I'll repeat this algorithm until convergence and try to obtain the global minimum (where θj will eventually converge)
Step 5: Testing
With the training complete and parameters learned, it’s time to test our model.
We’ll input x = 6 and see what our learned parameters return as the predicted output.
Result: 12.00 - perfect prediction
Step 6: Finding convergence
Let's find out when the model actually converged.
By tracking predictions every 50 epochs, I observed:
At epoch 750: Result = 11.999
At epoch 800: Result = 12.000
This confirms our model reached near-perfect convergence between 750 and 800 epochs and stayed stable afterward!
Full Code:
Conclusion
And that is how I coded a linear regression model from scratch!
CS229 felt very theoretical at the start, but applying the math hands-on helped me understand how the models actually learn.
Next up maybe I'll try logistic regression?
Stay updated for more!