Quick summary
Summarize this blog with AI
This deep learning reference connects architecture choices to a small executable network. The code uses NumPy to expose forward propagation, loss, gradients, and parameter updates. It is a teaching example on four synthetic XOR observations, not evidence that a model generalizes to new data.
Choose an architecture for the structure of the problem
| Architecture | Useful starting context | Important tradeoff |
|---|---|---|
| Multilayer perceptron | Fixed-length feature vectors | Does not inherently encode spatial or sequence structure |
| Convolutional network | Local spatial patterns, including images | Receptive field and representation choices affect captured context |
| Recurrent network | Sequential state updates | Sequential computation and long-range dependencies can be difficult |
| Transformer | Attention over token or feature sequences | Standard full attention forms a quadratic-size score matrix in sequence length |
| Autoencoder | Learning a representation through reconstruction | Low reconstruction error does not guarantee useful semantic features |
| Generative adversarial network | Generation through competing generator and discriminator objectives | Training stability and coverage of modes require care |
These are families, not exclusive task assignments. Compare against a simpler baseline and choose using data, compute, latency, and evaluation requirements.
Connect the forward pass, loss, and gradient
For a batch X, the network below computes H = tanh(XW₁ + b₁), logits Z = HW₂ + b₂, and probabilities sigmoid(Z). Binary cross-entropy is calculated directly from logits using logaddexp for numerical stability. Backpropagation applies the chain rule; gradient descent then subtracts a learning-rate-scaled gradient.
Weights start randomly so hidden units need not learn identical features. All gradients are calculated from the same forward pass before any parameters are updated. A single hidden layer is enough for this XOR demonstration; adding depth is not required to illustrate the training mechanism.
import numpy as np
X = np.array([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
y = np.array([[0.], [1.], [1.], [0.]])
rng = np.random.default_rng(7)
W1 = rng.normal(0, .5, (2, 8))
b1 = np.zeros((1, 8))
W2 = rng.normal(0, .5, (8, 1))
b2 = np.zeros((1, 1))
def forward():
hidden = np.tanh(X @ W1 + b1)
logits = hidden @ W2 + b2
probability = np.exp(-np.logaddexp(0, -logits))
loss = np.mean(np.logaddexp(0, logits) - y * logits)
return hidden, probability, loss
initial_loss = forward()[2]
for _ in range(3000):
hidden, probability, loss = forward()
dlogits = (probability - y) / len(X)
dW2 = hidden.T @ dlogits
db2 = dlogits.sum(axis=0, keepdims=True)
dhidden = (dlogits @ W2.T) * (1 - hidden**2)
dW1 = X.T @ dhidden
db1 = dhidden.sum(axis=0, keepdims=True)
W1 -= .1 * dW1
b1 -= .1 * db1
W2 -= .1 * dW2
b2 -= .1 * db2
_, probability, final_loss = forward()
print("predictions:", (probability >= .5).astype(int).ravel().tolist())
print("loss decreased:", bool(final_loss < initial_loss))
print("final loss below 0.05:", bool(final_loss < .05))
The expected predictions are [0, 1, 1, 0], and both checks print True. The model is evaluated on its training data only. Do not describe that result as test accuracy or use it to claim generalization.
Explain attention with dimensions
For a sequence of n tokens with query and key width d, Q and K have shape n×d. QKᵀ has shape n×n. Scaled dot-product attention computes softmax(QKᵀ/√d)V, normalizing each query’s scores across keys. A causal mask excludes future positions before normalization for autoregressive use. The original Transformer paper defines this mechanism.
The n×n score matrix explains an important full-attention memory cost; it does not describe every optimized implementation or sparse-attention variant. Multi-head attention uses several learned projections and combines their outputs. Positional information is needed when sequence order matters because content-only attention does not itself encode positions.
Distinguish training behavior from gradient tracking
Dropout randomly masks activations during training under its specified scaling rule and is disabled for ordinary evaluation. Batch normalization has different training and evaluation behavior involving batch statistics and stored estimates. A framework’s evaluation mode and its mechanism for disabling gradient tracking solve different problems; do not assume one automatically performs the other.
Keep preprocessing and class mappings consistent between training and inference. Store model and preprocessing versions together. For imbalanced outcomes, examine precision, recall, calibration, and task-specific error costs rather than accuracy alone. Tune thresholds on appropriate validation data, not on the final held-out evaluation set.
Debug in an order that reveals the problem
- Check tensor shapes, label encoding, finite values, and the loss/output pairing.
- Verify a tiny forward calculation and compare analytical gradients with finite differences on a small fixture.
- Try fitting a tiny training subset; failure can reveal an implementation or optimization problem.
- Inspect learning rate, gradient magnitudes, and training versus validation curves.
- Audit leakage, preprocessing differences, and the split before attributing results to architecture quality.
Overfitting is a gap between fit and generalization, not simply a large model. Regularization, data quality, augmentation, and model capacity can all matter. State the evidence behind a proposed fix and rerun an appropriate held-out evaluation.
