Tuning Fine-Tuning: Epochs, Learning Rate, and Batch Size

The situation

A team has a customisation job set up on Amazon Bedrock. They have a base model, a training dataset of a few hundred prompt-and-completion pairs that capture how their support replies should sound, and a smaller held-out set they kept back. They want the custom model to answer in the house style without being reminded in every prompt, and they want to stop paying for the long few-shot preamble they currently prepend to every call.

The first training run used the default hyperparameters and produced a model that mostly ignored the new style, sounding almost identical to the base. The second run cranked the number of passes over the data right up to force the behaviour in, and produced a model that reproduces the training replies almost word for word, drops in phrases that only made sense for the specific tickets it was trained on, and has quietly got worse at ordinary instruction-following it used to handle fine. Somewhere between those two runs is a model that learned the style and kept its general ability, and nobody wants to find it by launching twenty jobs and eyeballing the output.

The knobs are epochsEpochOne complete pass over the training dataset – more passes means more chance to shift behaviour, and more chance to memorise. , learning rateLearning rateHow far each training step moves the model’s weights – too low and nothing shifts, too high and it lurches past what you wanted. , and batch sizeBatch sizeHow many training examples the model sees before each weight update – mostly a stability and throughput dial, not a quality one. . The signal is the pair of loss curvesLoss curveThe plot of training error over time; the gap between the training and validation lines is how you spot memorising rather than learning. the job emits. The question is how to set the first from a reading of the second.

What actually matters

Fine-tuning is not a quality slider where more training is better. Each hyperparameter pushes the model in a direction, and past a point that depends on how much data you have, more of it makes the model worse in a way the training metrics can hide. The thing to hold onto is that the right amount of training is a function of the dataset, not a fixed recipe.

The dividing line that decides the most is underfitting versus overfitting. Underfitting is too little training: not enough passes over the data, or too gentle a learning rate, so the model never actually shifts its behaviour and you get something barely distinguishable from the base. Overfitting is too much: the model stops learning the general pattern in your data and starts memorising the specific examples, so it parrots training completions, latches onto surface quirks, and generalises badly to inputs it has not seen. Both are real failures, and they need opposite corrections, which is why guessing is expensive.

You tell the two apart by watching two curves, not one. Training loss measures error on the data the model is learning from; validation loss measures error on the held-out data it is not training on. When both fall together, the model is genuinely learning. When training loss keeps falling but validation loss flattens and then starts to rise, the model has begun memorising the training set at the expense of everything else, and that turning point is where overfitting starts. Training loss alone always looks like progress, because a model can always fit its own training data harder; the validation curve is what tells you when that progress has stopped being real.

The interaction that trips people up is dataset size. A small dataset overfits fast, because there is less variety to generalise from and the model runs out of genuine pattern to learn after only a pass or two, after which every further pass is memorisation. A large, varied dataset tolerates more passes before it turns. So the number of epochs is not a universal good number; it scales with how much data you have, and a few hundred examples wants noticeably fewer passes than tens of thousands.

Then there is the failure that does not show up in the style you were training for at all: catastrophic forgetting. Push the learning rate or the epoch count too hard and the model does not just overfit your task, it degrades on general capability it had before, because the aggressive updates overwrite weights that encoded skills you never meant to touch. This is why the training loss going down is not sufficient evidence of a good model, and why the honest test is a held-out evaluation against the base model on tasks you care about, not the loss number the job reports.

What we’ll filter on

1. Fit direction, is the model underfitting (barely changed from base) or overfitting (memorising training examples)?
2. Dataset size, how many passes can this much data support before it turns from learning to memorising?
3. Loss-curve signal, are training and validation loss falling together, or has validation flattened and started rising?
4. Stability, is the learning rate low enough to train smoothly rather than thrash, and high enough to actually move?
5. General capability retained, does the custom model still do the things the base could, or has it forgotten them?
6. Held-out result, does the model beat the base on an evaluation set, independent of what the loss number says?

The hyperparameter landscape

Epochs. An epoch is one complete pass over the training dataset. More epochs means the model sees each example more times and has more opportunity to shift its behaviour toward the data. Too few and it underfits: the behaviour never sets, and the custom model looks like the base. Too many and it overfits: after the model has extracted the general pattern, further passes just drive it to memorise specific completions, so validation loss turns up even as training loss keeps sinking. This is the knob most directly tied to dataset size; the right count for a few hundred examples is small, and it grows as the dataset does. On Bedrock the knobs all arrive the same way: CreateModelCustomizationJob takes a hyperParameters map of strings to strings, and the epoch count is epochCount in that map, with a default and a permitted range that belong to the base model rather than to the service. So the sensible starting move is the default for your base model, then adjust from what the loss curves show.

Learning rate (and the learning-rate multiplier). The learning rate governs how large a step each update takes toward the training data. Most base models take it as learningRate, an absolute value; some expose learningRateMultiplier instead, scaling the model’s own tuned base rate, which is why the number you set is sometimes a small multiplier rather than a raw rate. Set it too high and training destabilises: the loss jumps around instead of descending smoothly, updates overshoot, and you invite catastrophic forgetting because the model is making violent changes to its weights. Set it too low and the model learns too slowly, so within a reasonable number of epochs it never reaches the behaviour you wanted, which reads as underfitting even though the real problem is timid steps. It trades off against epochs: a lower rate needs more passes to arrive, a higher rate arrives faster but risks blowing past a good state.

Batch size. Batch size is how many examples the job processes before it updates the model once. Larger batches average over more examples per update, which makes each step smoother and more stable and improves throughput, at the cost of memory and sometimes a model that generalises slightly less well. Smaller batches update more often on noisier estimates, which can help the model escape a rut but makes training less steady. It is usually the knob you touch last; get epochs and learning rate roughly right first, and treat batch size as a stability and throughput adjustment rather than the main lever on quality. On Bedrock it is batchSize, and it too has a base-model-specific default and range, narrow enough on some models that there is nothing to tune.

The loss curves as the reading instrument. These are not a hyperparameter you set; they are the output you steer by. Training loss falling means the model is fitting the data it sees. Validation loss, computed on held-out data the model does not train on, is the one that tells you whether the fit is generalising. The pattern to recognise: both falling is healthy; validation flattening while training keeps falling is the onset of overfitting; validation rising while training still falls is overfitting in progress. The point just before validation turns is the best model the run produced, which is the whole idea behind stopping early rather than always running every epoch you scheduled.

The curves are files, not a dashboard, and knowing where they land is half of using them. Bedrock writes them to the S3 prefix you gave the job as its outputDataConfig, under a folder named for the training job: training_artifacts/step_wise_training_metrics.csv carries a row per step with the step number, the epoch number, the training loss, and perplexity, and validation_artifacts/post_fine_tuning_validation/validation_metrics.csv carries the same columns with validation loss in place of training loss. Plotting one against the other is how the run gets read. Summary figures also come back in the trainingMetrics and validationMetrics fields of a GetModelCustomizationJob response, but the per-step CSVs are what show you the shape, and the shape is the whole signal.

Early stopping. Rather than committing to a fixed epoch count and taking whatever comes out, early stopping watches validation loss and halts when it stops improving for a while, keeping the model from the best point instead of the last point. It is the direct operational answer to overfitting: it caps training at the moment the curves say further passes would only memorise. Where the base model supports it, the job takes it as two more hyperparameters, earlyStoppingThreshold for how much improvement counts as improvement and earlyStoppingPatience for how many rounds of no improvement to tolerate before halting, which turns “how many epochs” from a guess into something the validation curve decides for you.

Side by side

Symptom	Epochs	Learning rate	Batch size	Loss-curve tell	Correction
Model unchanged from base	Too few	Too low	✗	Both curves stay high, barely fall	More epochs, or a higher rate
Learns but thrashes	✗	Too high	Too small	Loss jumps around, no smooth descent	Lower the rate, raise batch size
Parrots training replies	Too many	Too high	✗	Training falls, validation flattens then rises	Fewer epochs, stop early
Forgot general skills	Too many	Too high	✗	Good training loss, poor held-out eval	Fewer epochs, lower rate, re-evaluate
Learning too slowly	Too few	Too low	✗	Both curves fall, never reach a floor	Higher rate, or more epochs
Healthy run	Right for the data	Stable	Steady	Both fall together, level off	Stop at the validation low

The picks in depth

Start from the base model’s default hyperparameters, because they are picked to be a reasonable centre for that model rather than an arbitrary guess, and change one thing at a time from what the curves tell you. The first run in the situation, the one that came out looking like the base, is textbook underfitting: not enough passes over a few hundred examples for the style to set, so the correction is more epochs, and if it still will not move, a modestly higher learning rate so each pass shifts the model further. The second run, the parrot, is the opposite failure from pushing the epoch count too far for that small a dataset, so the fix is fewer passes, not more forceful ones, and this is exactly where the validation curve earns its place: the good model was somewhere in the middle of that long run, at the point validation loss bottomed out, and stopping there rather than at the final epoch would have caught it.

Read the epoch count against the dataset first. A few hundred examples is a small dataset, and small datasets turn from learning to memorising after only a couple of passes, so the instinct to crank epochs up to force a stubborn behaviour is precisely wrong; it is the move that produced the parrot. If the behaviour will not set at a sensible epoch count, the lever to reach for is the learning rate or better data, not ten more passes over the same few hundred rows. When the dataset grows into the thousands or tens of thousands, more epochs become safe and often necessary, because there is enough variety that each pass is still teaching a general pattern rather than drilling specific rows.

Treat the learning rate as the stability control. If the loss will not descend smoothly and jumps around between steps, the rate is too high; bring it down and the descent steadies. If the model learns cleanly but too gradually to arrive within your epoch budget, the rate is too low; nudge it up. On the base models that expose a learning-rate multiplier rather than a raw rate, the same logic holds, the multiplier is just scaling the model’s own base rate. The rate is also the knob most implicated in catastrophic forgetting, so when a model trains to a nice training loss but flunks a held-out check of its old general ability, suspect too high a rate driving updates that overwrote skills you meant to keep.

Leave batch size until epochs and rate are roughly right, then use it to smooth or speed the run. A larger batch gives steadier updates and better throughput and is the natural response to a noisy, thrashing training curve once you have already checked the learning rate; a smaller batch updates more often and can help a stalled run, at the cost of steadiness. It is a supporting adjustment, not the main quality lever, and it is rarely where a bad custom model went wrong.

Whatever the loss curves say, the model that ships is decided by a held-out evaluation, not the loss number. Run the custom model and the base model against the evaluation set you kept back, on the task you actually care about, and compare. The loss curve tells you when the run was healthy; the evaluation tells you whether the result is better than what you started with and whether it kept the general ability you needed. A model with a beautiful training loss that loses to the base on held-out data is not a good model, and only the evaluation reveals that.

A worked example: finding the epochs for a few hundred examples

The dataset is three hundred prompt-and-completion pairs of house-style support replies, with sixty more held back as a validation and evaluation set. The base model’s default epoch count is the starting point.

The first job runs at the default and comes out sounding like the base. The training and validation loss both fell a little and then flattened high, the picture of underfitting; the model did not see the data enough times to shift. The correction is to raise the epoch count and rerun, watching the curves rather than the output alone.

The next job runs with the epoch count pushed well up to force the point. This time training loss keeps sinking to a low floor, but validation loss falls, bottoms out partway through, and then climbs for the back half of the run. The model in hand at the final epoch is the parrot: it reproduces training replies verbatim and has started failing on ordinary requests it used to handle. Reading the validation curve, the best model was at its low point, not at the end, which is what early stopping would have kept. So the corrected run either sets the epoch count near that turning point or enables early stopping so the job halts when validation stops improving. The learning rate stays at the default, because the descent was smooth, no thrashing, so the problem was passes, not step size.

The final check is not the loss at all. The chosen custom model and the base model both run against the sixty held-out pairs; the custom model matches the house style and still handles the general requests the base did, and it wins the comparison. That is the evidence the model is ready, and it is evidence the loss curve alone could not have given, because a lower training loss and a better model are not the same claim. The technique underneath this is the same discipline as matching a prompt technique to the task shape: the setting that helped one job is not a universal good, and the right value is read from the data in front of you.

What’s worth remembering

1. The two failure modes are opposite: underfitting is too little training and a model that looks like the base; overfitting is too much and a model that memorises the training data and generalises badly.
2. Epochs are passes over the data; too few underfits, too many overfits, and the right number scales with dataset size rather than being a fixed recipe.
3. A small dataset overfits fast, so it wants fewer epochs; cranking epochs up to force a stubborn behaviour on little data is how you get a model that parrots its training set.
4. Learning rate sets step size: too high destabilises training and invites forgetting, too low learns so slowly the behaviour never sets within a sensible number of passes.
5. Some Bedrock base models expose a learning-rate multiplier that scales the model’s base rate rather than an absolute value; the same high-and-low logic applies.
6. Batch size trades throughput and stability against memory and generalisation; set it last, as a steadiness adjustment once epochs and rate are roughly right.
7. Watch training and validation loss together: both falling is healthy, but training falling while validation flattens then rises is the onset of overfitting.
8. Early stopping keeps the model from the point validation loss bottomed out rather than the last epoch, turning “how many epochs” from a guess into something the curve decides.
9. Overtuning can cause catastrophic forgetting, degrading general capability the base model had, which a good training loss will hide.
10. The model that ships is chosen by a held-out evaluation against the base model, not by the loss number, because a lower training loss and a better model are not the same thing.