PyTorch & Practical Model Training

Start with a training loop you can explain#

Before adapting a language model, build a small program whose inputs, outputs, gradients, and checkpoints you understand. PyTorch supplies tensor operations, automatic differentiation, neural-network modules, and optimizers. Those pieces are powerful precisely because you can inspect them separately.

This article follows one CPU-only regression example. It generates synthetic two-feature inputs and targets from a known equation, fits a small model, evaluates held-out data, saves its state, and verifies that loading reproduces predictions. It needs no downloaded dataset, GPU, Hugging Face account, or model weights.

The example is educational software, not a Makata language-model experiment. Its loss values measure a synthetic regression problem. They must not be presented as evidence about Filipino or Taglish capability.

Download the complete Python example. The excerpts below are selected from that implementation, with smaller standalone snippets used to explain tensors and autograd. The full file is the executable source of truth. The official PyTorch fundamentals tutorials offer a complementary introduction.

Tensors and tensor operations#

A tensor is an array with a shape, data type, and device. Shape tells you how dimensions are arranged; data type tells you how values are stored; device tells you where operations run. Our inputs use floating-point tensors on the CPU.

import torch

inputs = torch.tensor([[1.0, 2.0], [-1.0, 0.5]])
weights = torch.tensor([[3.0], [-2.0]])
predictions = inputs @ weights + 0.5
print(predictions)
# tensor([[-0.5000],
#         [-3.5000]])

The input shape is (2, 2): two examples with two features each. The weight shape is (2, 1). Matrix multiplication produces (2, 1), one output per example. The scalar bias broadcasts to both rows. The at-sign operator performs matrix multiplication; an asterisk would perform elementwise multiplication and therefore express a different operation.

Floating-point rounding means decimal arithmetic is approximate. Compare numerical results with an appropriate tolerance unless you specifically expect an identical save/load round trip. Check data types deliberately: integer token IDs and floating-point model weights serve different roles. Moving a model to a device does not automatically move an unrelated input tensor there.

The official tensor tutorial covers indexing, joining, reshaping, and device movement. For this example, the important habit is to write expected shapes before an operation and inspect actual shapes afterward.

Broadcasting can conceal a bug

Suppose predictions have shape (32, 1) but targets have shape (32,). Subtracting them can broadcast into (32, 32), comparing every prediction against many targets instead of pairing examples. The operation may run while optimizing the wrong quantity.

Our example keeps targets shaped (N, 1) and explicitly checks that predictions and targets match. Do not remove dimensions simply to silence an error without understanding the resulting shape. A batch with one example is particularly useful for testing because indiscriminate squeezing can accidentally remove its batch dimension.

Reshaping also does not reorder axes in the same way as a transpose. A token tensor with batch, sequence, and feature axes must preserve the intended example-token associations. Shape correctness is necessary, but knowing what each axis represents is equally important.

Computational graphs and automatic differentiation#

When an operation uses a tensor requiring gradients, PyTorch can record the operation in a computational graph. The graph connects outputs to the earlier values needed to compute derivatives. A backward call propagates derivatives through this graph.

import torch

w = torch.tensor(1.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
prediction = w * 2.0 + b
loss = (prediction - 5.0).square()
loss.backward()
print(w.grad, b.grad)
# tensor(-12.) tensor(-6.)

The prediction is $2$, the error is $-3$, and the squared loss is $9$. The derivative with respect to the prediction is $-6$. Multiplying by the input $2$ gives the weight gradient $-12$; multiplying by the bias's local derivative $1$ gives $-6$.

These are analytical derivatives of the operations, not random estimates. Automatic differentiation automates the chain rule; it does not guess a favorable update. The autograd tutorial explains the relationship between recorded operations and gradient tensors.

Step through the graph and distinguish values from derivatives. A large forward value does not imply a large gradient. The gradient depends on the entire path from that intermediate quantity to the loss.

Gradient accumulation and graph lifetime

By default, backward contributions accumulate in parameter gradient fields. This supports intentional gradient accumulation across microbatches. In a basic one-batch-per-update loop, clear old gradients before computing the new batch's gradients.

Calling optimizer.zero_grad(set_to_none=True) marks gradients absent rather than filling existing tensors with zeros. The next backward pass populates them. A parameter with no gradient is not always equivalent to a parameter whose gradient happens to be numerically zero; optimizers can treat the cases differently.

Normally, saved intermediate values needed for backward are released after a backward call. Reusing the same graph for another backward pass can require retaining it, but retaining every graph is not a standard training fix. Usually, a new forward pass should construct the next graph. Accumulating tensor losses in a Python list without detaching them can also retain graphs and consume memory.

Defining architecture with nn.Module#

An nn.Module packages a model's computation and registered parameters. Our model is intentionally small:

from torch import Tensor, nn

class Regressor(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.linear = nn.Linear(2, 1)

    def forward(self, inputs: Tensor) -> Tensor:
        return self.linear(inputs)

Calling the parent initializer establishes PyTorch's module bookkeeping. Assigning nn.Linear as an attribute registers its weight and bias, making them visible to model.parameters() and model.state_dict(). The layer has two input features and one output feature, so it contains two learned weights and one learned bias.

The forward method describes the calculation. Calling model(inputs) uses the module's call machinery and then invokes this method. Prefer that form over directly calling model.forward(inputs), which can bypass hooks and other module behavior.

Our ground-truth relationship is linear, so a linear model is sufficient. Adding several hidden layers would complicate a problem already representable by three parameters. The official module tutorial shows how the same mechanism composes larger networks. Simplicity here makes training behavior easier to diagnose.

Generating and partitioning data#

The target rule is

where $\epsilon$ is small independent noise. The script generates 512 examples using a dedicated random generator. It splits them into 384 training, 64 validation, and 64 test examples using a different seeded generator.

generator = torch.Generator().manual_seed(41)
inputs = torch.randn(512, 2, generator=generator)
noise = 0.1 * torch.randn(512, 1, generator=generator)
targets = inputs @ torch.tensor([[3.0], [-2.0]]) + 0.5 + noise

Using separate generators isolates data creation from model initialization and batch ordering. Adding a random operation in one part of the program is then less likely to silently change every other source of randomness. Seeds make this controlled example repeatable within a defined environment; they do not guarantee bitwise equality across arbitrary hardware or library versions.

We know the target rule because we created it, but the model only receives inputs and targets. The model should approach weights near $3$ and $-2$ with bias near $0.5$, subject to finite data, noise, and optimization. Those values are an interpretable diagnostic, not a replacement for validation.

For real text data, a random row split is often insufficient because related or duplicated documents can leak across partitions. This synthetic setup has independent generated examples. Do not generalize its splitting strategy blindly to a Philippine-language corpus.

Batching with Dataset and DataLoader#

A Dataset describes how examples are accessed. A DataLoader groups examples into batches and can control shuffling and worker processes. TensorDataset is enough when inputs and targets already live in tensors.

loader_generator = torch.Generator().manual_seed(43)
training_loader = DataLoader(
    training, batch_size=32, shuffle=True, generator=loader_generator,
    num_workers=0,
)

Each yielded pair has input shape (32, 2) and target shape (32, 1). With 384 training examples and batch size 32, one epoch contains 12 optimizer updates. An epoch is a pass through the training dataset; a step is one optimizer update. These terms are not interchangeable.

Shuffling changes the sequence of updates while preserving which examples belong to training. Validation does not need shuffling because we only aggregate predictions. We use zero worker processes to keep the small CPU example straightforward and portable, especially on systems where worker startup requires special entry-point handling.

The DataLoader tutorial describes more flexible datasets and loading behavior. More workers can help when preprocessing or input/output is a bottleneck; they are not a universal speed improvement for a tiny in-memory dataset.

Optimizers and the training loop#

The script uses AdamW with learning rate $0.03$ and weight decay $0$. Zero decay keeps this educational regression objective focused on fitting the known relationship. It is not a recommended default for every language-model run.

optimizer = torch.optim.AdamW(model.parameters(), lr=0.03, weight_decay=0.0)

The optimizer receives references to registered parameters. It does not need the dataset directly. It reads each parameter's gradient when its update method runs.

The central batch operation is:

optimizer.zero_grad(set_to_none=True)
predictions = model(inputs)
if predictions.shape != targets.shape:
    raise ValueError("Prediction and target shapes must match.")
loss = loss_fn(predictions, targets)
if not torch.isfinite(loss):
    raise RuntimeError("Non-finite training loss.")
loss.backward()
optimizer.step()

Read these lines as a data flow. Inputs pass through the current model. Predictions and targets produce a scalar loss. Backward fills gradients. The optimizer uses those gradients to update parameters. The next batch starts from the updated values.

The finite-loss check stops this example when arithmetic produces an infinity or a not-a-number value. Continuing from corrupted updates usually obscures the original cause. For larger systems, you may additionally inspect gradient finiteness, mixed-precision scaling, and the first operation that produces an invalid value.

The official optimization tutorial documents this separation between loss, backward propagation, and optimizer updates. Our program adds explicit shapes, reproducible data, and checkpoint checks so the loop can be inspected as a complete experiment.

Averaging the epoch loss correctly

A mean loss from a small final batch should not receive the same weight as a mean from a full batch when reporting per-target MSE. The script multiplies each batch mean by its number of target elements, sums those contributions, and divides by the total element count.

For example, a batch of 32 targets with MSE $2$ and a batch of eight targets with MSE $6$ have combined MSE $(32\cdot2+8\cdot6)/40=2.8$. Averaging the two batch means would incorrectly report $4$.

Training loss also uses predictions from successive parameter states within the epoch. Validation at epoch end uses one fixed state. Their difference can therefore reflect both data generalization and measurement timing, even before considering dropout or other training-only behavior.

The browser explorer calculates a small deterministic learning process. Its curve illustrates optimization behavior and is separate from the downloadable script's saved measurements. Neither curve is an LLM benchmark.

Validation without parameter updates#

Validation asks how the current model predicts held-out examples. It should not update weights or retain unnecessary backward graphs:

@torch.no_grad()
def evaluate(model: Regressor, loader: DataLoader[tuple[Tensor, ...]]) -> float:
    model.eval()
    squared_error = 0.0
    count = 0
    for inputs, targets in loader:
        predictions = model(inputs)
        squared_error += (predictions - targets).square().sum().item()
        count += targets.numel()
    return squared_error / count

The decorator disables gradient recording within this function. model.eval() separately sets evaluation behavior for modules such as dropout. Our linear layer behaves the same in either mode, but making the distinction explicit prevents errors when the architecture later changes.

The training function calls model.train() at the next epoch. Forgetting to restore it can leave a more complex model in evaluation mode during training. Conversely, calling evaluation mode without disabling gradients can waste memory even when no optimizer update occurs.

We select the checkpoint with the lowest validation MSE and evaluate that model on the test set only after training. If you repeatedly change the procedure based on the final test result, the test is no longer an untouched assessment. The split labels alone cannot enforce disciplined methodology.

Saving and loading checkpoints#

A state dictionary maps parameter and buffer names to tensors. Saving it separates numerical state from the Python class definition. You still need compatible architecture code to reconstruct the model.

For continued training, model weights alone are insufficient. The optimizer keeps running statistics, the data loader has a random ordering state, and the program needs to know which epoch it has completed. The example's checkpoint includes those items, the best validation value, best model state, history, and runtime versions.

checkpoint = torch.load(resume, map_location="cpu", weights_only=True)
model.load_state_dict(checkpoint["model"])
optimizer.load_state_dict(checkpoint["optimizer"])
loader_generator.set_state(checkpoint["loader_rng"])
torch.set_rng_state(checkpoint["torch_rng"])

The file is loaded explicitly onto the CPU. Restricted loading with weights_only=True supports the tensors and primitive structures used here and avoids requiring a serialized Python model object. Treat downloaded checkpoints as external files whose origin matters. Consult PyTorch's serialization documentation when using more complex checkpoint formats.

The program writes last.pt for resuming the latest epoch and best-weights.pt for the validation-selected weights. These are distinct purposes. Resuming from a best checkpoint while reusing the latest optimizer state would combine inconsistent states.

The best model is stored using a deep copy of the state dictionary. Merely assigning the live dictionary to another variable can leave its tensors sharing storage with the model, allowing later updates to overwrite what you thought was the saved best state.

What a round-trip check establishes

After saving, the script builds a new model, loads the latest weights, and compares its predictions with those of the model that was saved. It requires exact equality for this CPU round trip. This verifies that the current architecture and serialization path preserve outputs.

It does not prove that an arbitrary future library version will behave identically, that the training data was valid, or that the model generalizes. Checkpoint integrity is one test within a wider verification process. For a real research run, also preserve configuration, dataset identifiers, tokenizer revision, and source revision.

Running the example#

Create a Python environment outside the website dependency tree. Install a CPU build of PyTorch using the official installation selector for your operating system. This downloadable script was verified using Python 3.12.14 and PyTorch 2.14.1+cpu; those versions describe the verification environment, not a claim that other supported versions cannot run it.

After downloading the file, execute:

python train_regression.py --epochs 80 --output regression-output

The script prints a JSON summary and creates a training history, result summary, latest checkpoint, and best weights. Outputs should be finite. Learned coefficients should be close to the synthetic rule rather than arbitrary, and the checkpoint round-trip flag should be true.

To examine resumption, run a shorter training segment and continue to a total of 80 epochs:

python train_regression.py --epochs 40 --output regression-partial
python train_regression.py --epochs 80 --resume regression-partial/last.pt --output regression-resumed

The epoch argument is the total desired epoch count, not the number of additional epochs. The saved loader generator preserves future shuffle order. Checkpoints are written at epoch boundaries; this program does not promise exact recovery halfway through a partially consumed batch sequence.

The example deliberately writes outputs to the directory you choose. Run it in a scratch directory and keep generated checkpoints out of the public website repository. It is a learning artifact, not the beginning of Makata's production training infrastructure.

Debugging training failures#

When loss does not fall, inspect the smallest explanation first. Print one batch's shapes, data types, target range, predictions, and loss. Verify that model parameters require gradients, that their gradients are populated after backward, and that the optimizer references the same parameters used in the forward pass.

A decreasing loss can still accompany a bug. Compare against the known target rule, test a tiny batch, and inspect held-out performance. If the model predicts the same value for every input, consider both a legitimate bias-only solution and a broken input path. If gradients are absent, look for detached tensors, conversion to ordinary Python numbers, or operations outside gradient tracking.

For non-finite values, find the first bad quantity rather than merely lowering the learning rate until the symptom disappears. Common causes include division by zero, logarithms of zero, excessive update scales, and invalid data. Normalizing features can improve numerical behavior, but the normalization statistics must come from training data and then be applied consistently to held-out data.

Memory growth over epochs often points to retained computation graphs or stored outputs that still require gradients. Logging a scalar with loss.item() records a Python number. Saving detached predictions is different from storing tensors that keep their whole history alive.

Inspecting one update before running many

A focused debugging exercise is to stop after one batch and compare a parameter before and after its update. Save a detached clone of the weight tensor, run forward and backward, inspect the gradient, apply the optimizer, and inspect the difference. A clone matters because another reference to the same tensor will reflect the updated storage.

For basic stochastic gradient descent without momentum or weight decay, the difference should equal negative learning rate times gradient. For AdamW, the relationship includes moving statistics and adaptive scaling, so do not expect that simpler identity. First test the algorithm you actually configured.

Also check that the intended parameters are registered. A tensor stored as an ordinary attribute is not automatically a trainable parameter merely because it contains floating-point values. Built-in layers register their own parameters. If you create a custom learned tensor, use the framework's parameter mechanism and confirm that it appears in the optimizer's parameter list.

Freezing a parameter means preventing its update under the chosen procedure. Setting its gradient requirement to false stops gradient accumulation for that tensor, but the forward operation may still participate in a larger graph needed by other trainable components. This distinction becomes important when adapters are inserted into a frozen language model.

Interpreting a training history

The history file records training and validation MSE per epoch. Look at its trajectory rather than just the final entry. A rapid initial improvement followed by a plateau is plausible for this simple task. A training loss that improves while validation degrades may indicate overfitting or a mismatch between the two datasets.

The synthetic target includes noise, so driving validation loss exactly to zero is neither expected nor necessary. The model does not know the individual unseen noise values. It can learn the underlying linear relationship, but irreducible uncertainty remains in targets generated with new noise.

If you extend the script to standardize inputs, calculate the mean and standard deviation from the training partition only. Save those statistics with the model and apply them identically at validation and inference. A correct model paired with different preprocessing computes a different overall prediction system.

When reporting an experiment, preserve the complete history and state how a checkpoint was selected. Choosing the best observed validation epoch is itself a selection procedure. It is more informative to document that rule than to show only whichever curve segment looks smoothest.

Reproducibility has a stated boundary

The program fixes model initialization, synthetic data, split selection, and training shuffle order. It uses one CPU thread and asks PyTorch for deterministic algorithms. These choices define a small, controlled verification setting; they are not a guarantee about every device, operation, or future library release.

Resuming a general training system can also require learning-rate scheduler state, automatic mixed-precision scaler state, distributed sampler position, augmentation generators, and worker state. Those are absent from this example because it does not use those features. Adding them later should include an explicit resume test rather than assuming the current checkpoint dictionary covers them automatically.

A useful test compares an uninterrupted run with a run interrupted at a documented boundary and resumed to the same total number of steps. Compare parameter tensors, optimizer state, and the metric history. A mismatch should prompt investigation of state restoration before drawing conclusions about stochastic model behavior.

From regression to language-model training#

The control flow remains recognizable when the model becomes a transformer. Inputs become token IDs and masks. Targets become shifted token IDs. Cross-entropy replaces regression MSE. The architecture and optimizer state become much larger, making GPU memory, precision, and checkpoint efficiency important.

Data preparation also becomes more demanding. You must decide which tokens contribute to loss, how examples are packed, how padding is masked, and how held-out text stays uncontaminated. Those decisions determine the meaning of the training signal as directly as the optimizer code does.

For Makata, the immediate benefit is a disciplined workflow: establish a baseline, record configuration, validate the implementation on a small case, and make one controlled change. A working loop is a prerequisite for meaningful experiments, not evidence that a Philippine-language model has already been trained.

Common misconceptions#

• A GPU is required to learn PyTorch. Small tensor operations and training examples work on a CPU.
• Backward updates the model. It computes gradients; the optimizer applies updates.
• Evaluation mode and no-gradient mode are interchangeable. They control different behavior and are often used together.
• Saving weights is enough to resume exactly. Optimizer statistics, random states, and position in the data stream also matter.
• Matching shapes proves the loss is correct. Targets, masking, reduction, and example alignment must also match the intended task.

Key takeaways#

Read training as a sequence of inspectable operations: load a batch, calculate predictions, calculate loss, clear and compute gradients, update parameters, and evaluate held-out data. Preserve the state required to resume that sequence, and verify loading by comparing outputs.

Use this example to become comfortable with execution and debugging before increasing model size. Continue with transformers for language-model architecture, dataset engineering for reliable inputs, and GPU computing for the resources larger models require.