What is a neural network?

A neural network is a function approximator made of layers of weighted connections. Weights are adjusted during training to map inputs to correct outputs. All modern AI including LLMs are built from neural networks.

AIMachine Learning

Overview

A neural network is a mathematical system loosely inspired by the brain. It consists of layers of nodes (neurons) connected by weighted edges. During training, input data flows forward through the network, a loss function measures how wrong the prediction was, and backpropagation adjusts the weights to reduce the error. Repeat millions of times and the network learns to make accurate predictions.

Key Points

•A neuron computes a weighted sum of its inputs plus a bias, then passes the result through an activation function (ReLU, sigmoid).
•Input layer receives raw data (pixels, token embeddings). Hidden layers transform it. Output layer produces the prediction.
•Training uses gradient descent: compute the gradient of the loss with respect to every weight and nudge each weight in the direction that reduces loss.
•Backpropagation efficiently computes all gradients in one backward pass using the chain rule of calculus.
•Universal approximation theorem: a network with one hidden layer and enough neurons can approximate any continuous function.
•Modern LLMs are neural networks with hundreds of billions of parameters trained on trillions of tokens.

Examples

A single neuron and a tiny 2-layer network from scratch

python

import numpy as np

def relu(x):
    return np.maximum(0, x)

def relu_gradient(x):
    return (x > 0).astype(float)

class TinyNeuralNet:
    def __init__(self, input_size, hidden_size, output_size):
        # Xavier initialisation
        self.W1 = np.random.randn(input_size, hidden_size) * 0.01
        self.b1 = np.zeros(hidden_size)
        self.W2 = np.random.randn(hidden_size, output_size) * 0.01
        self.b2 = np.zeros(output_size)

    def forward(self, X):
        # Layer 1: linear transform then activation
        self.z1 = X @ self.W1 + self.b1   # (N, hidden_size)
        self.a1 = relu(self.z1)             # (N, hidden_size)

        # Layer 2: linear transform (output, no activation)
        self.z2 = self.a1 @ self.W2 + self.b2  # (N, output_size)
        return self.z2

    def backward(self, X, y, learning_rate=0.01):
        N = X.shape[0]

        # Loss gradient at output (MSE: d(loss)/d(z2))
        dz2 = 2 * (self.z2 - y) / N          # (N, output_size)

        # Gradients for W2, b2
        dW2 = self.a1.T @ dz2                 # (hidden_size, output_size)
        db2 = dz2.sum(axis=0)

        # Propagate back through ReLU
        da1 = dz2 @ self.W2.T                 # (N, hidden_size)
        dz1 = da1 * relu_gradient(self.z1)    # element-wise

        # Gradients for W1, b1
        dW1 = X.T @ dz1
        db1 = dz1.sum(axis=0)

        # Update weights (gradient descent step)
        self.W1 -= learning_rate * dW1
        self.b1 -= learning_rate * db1
        self.W2 -= learning_rate * dW2
        self.b2 -= learning_rate * db2

# Training loop
net = TinyNeuralNet(input_size=2, hidden_size=4, output_size=1)
X = np.array([[0,0],[0,1],[1,0],[1,1]])  # XOR inputs
y = np.array([[0],[1],[1],[0]])           # XOR labels

for epoch in range(1000):
    predictions = net.forward(X)
    net.backward(X, y, learning_rate=0.1)

print(net.forward(X).round(2))  # [[0.], [1.], [1.], [0.]]

The same network in PyTorch (modern approach)

python

import torch
import torch.nn as nn

class TinyNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Linear(2, 4),
            nn.ReLU(),
            nn.Linear(4, 1),
        )

    def forward(self, x):
        return self.layers(x)

model     = TinyNet()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
criterion = nn.MSELoss()

X = torch.tensor([[0,0],[0,1],[1,0],[1,1]], dtype=torch.float32)
y = torch.tensor([[0],[1],[1],[0]], dtype=torch.float32)

for epoch in range(1000):
    pred  = model(X)              # forward pass
    loss  = criterion(pred, y)    # compute loss
    optimizer.zero_grad()         # clear old gradients
    loss.backward()               # backpropagation
    optimizer.step()              # update weights

print(model(X).detach().round())  # tensor([[0.],[1.],[1.],[0.]])

PyTorch (and JAX, TensorFlow) compute backpropagation automatically - you only write the forward pass.

Pro Tip

You do not need to implement backpropagation manually to use neural networks - frameworks like PyTorch and TensorFlow handle it automatically. Understanding the concept (forward pass computes predictions, backward pass computes gradients, optimizer updates weights) is enough to use and debug most models effectively.