The situation
A team has a customisation job set up on Amazon Bedrock. They have a base model, a training dataset of a few hundred prompt-and-completion pairs that capture how their support replies should sound, and a smaller held-out set they kept back. They want the custom model to answer in the house style without being reminded in every prompt. They also want to stop paying for the long few-shot preamble they currently prepend to every call.
The first run used the default hyperparameters. It produced a model that sounded almost identical to the base, with little of the new style in it. The second run pushed the number of passes over the data to the model’s maximum. That model reproduces the training replies almost word for word, drops in phrases that only made sense for the specific tickets it was trained on, and has got worse at ordinary instruction-following it previously handled. Somewhere between those two runs is a model that learned the style and kept its general ability. Launching job after job and eyeballing the output is a slow and expensive way to find it.
The knobs are EpochOne complete pass over the training dataset – more passes means more chance to shift behaviour, and more chance to memorise., Learning rateHow far each training step moves the model’s weights – too low and nothing shifts, too high and it lurches past what you wanted., and Batch sizeHow many training examples the model sees before each weight update – mostly a stability and throughput dial, not a quality one.. The signal is the pair of Loss curveThe plot of training error over time; the gap between the training and validation lines is how you spot memorising rather than learning. the job emits. The work is setting the first from a reading of the second.
What actually matters
Fine-tuning is not a quality slider where more training is better. Each hyperparameter moves the model in a direction, and past some point more of it produces a worse model in a way the training metrics hide. The search space is also smaller than the framing suggests. On most base models the epoch count tops out in single digits, so there are a handful of values to try, not dozens.
Underfitting and overfitting are the two directions, and AWS defines both. A model underfits when it performs poorly on the training set and lacks the capacity to learn it. A model overfits when it performs well on the training data and poorly on the validation data. Both are real failures, and they need opposite corrections. That is why the curves are worth reading before the next run is launched.
You tell the two apart by watching two curves rather than one. Training loss measures error on the data the model is learning from. Validation loss measures error on the held-out data it is not training on. Both falling together means the model is learning. Training loss falling while validation loss flattens and then rises means the run has crossed into overfitting. Training loss on its own always looks like progress, because a model can always fit its own training data harder.
Dataset size is where most intuitions go wrong. The folk rule says a small dataset needs fewer passes, and AWS publishes the opposite: larger datasets require fewer epochs to converge, and smaller datasets require more. That guidance appears on the Bedrock hyperparameter reference and again in the Nova fine-tuning documentation. Fewer examples means fewer weight updates per pass, so convergence takes more passes to reach. Epoch count does depend on your data, but a row count does not predict the stopping point. The validation curve does. There is a separate failure that never shows up in the style you were training for: training hard on one task can degrade the model on work it already handled. AWS approaches this from the other side when it recommends LoRA adapters for their regularising effect, which reduces overfitting and the risk of the model losing the source domain. A managed Bedrock fine-tuning job does not expose that dial, so a falling training loss is not sufficient evidence of a good model.
What we’ll filter on
- Fit direction, is the model underfitting (barely changed from base) or overfitting (reproducing training examples)?
- Convergence, has the run had enough passes to converge, bearing in mind that a smaller dataset needs more rather than fewer?
- Loss-curve signal, are training and validation loss falling together, or has validation flattened and started rising?
- Stability, is the learning rate low enough to descend smoothly and high enough to move the model at all?
- General capability retained, does the custom model still do the things the base could?
- Held-out result, does the model beat the base on an evaluation set, independent of what the loss number says?
The landscape
Epochs. An epoch is one complete pass over the training dataset. More epochs means the model sees each example more times. Too few and it underfits, so the behaviour never sets and the custom model resembles the base. Too many and it overfits, so validation loss turns up while training loss keeps sinking. On Bedrock the knobs arrive through one API. CreateModelCustomizationJob takes a hyperParameters field typed as a string-to-string map, and the epoch count is epochCount in that map. The default and permitted range belong to the base model rather than to the service. Amazon Nova Micro, Lite and Pro allow 1 to 5 and default to 2. Anthropic Claude 3 models allow 1 to 10 and default to 2. Meta Llama 3.1 and 3.2 allow 1 to 10 and default to 5. Cohere Command allows 1 to 100 on an adjustable quota and defaults to 1. Start at the default for your base model and adjust from what the curves show. Check the model’s lifecycle state first: once a model enters the Legacy state on Bedrock, no new fine-tuning jobs can be created against it.
Learning rate (and the learning-rate multiplier). The learning rate sets how large a step each update takes toward the training data. Most base models take learningRate, an absolute value, which on the Nova understanding models runs from 1e-6 to 1e-4 and defaults to 1e-5. Anthropic Claude 3 models expose learningRateMultiplier instead, from 0.1 to 2 with a default of 1, scaling the model’s own tuned base rate. Set it too high and training destabilises: the loss jumps around instead of descending, and updates overshoot. AWS is explicit that a higher rate can reach convergence faster but is the less desirable route, because it can cause training instability at convergence. Set it too low and the model learns too slowly to arrive within the epoch budget, which reads as underfitting even though the cause is small steps.
The Nova and Titan text models add a third knob, learningRateWarmupSteps, the number of iterations over which the rate climbs to the value you set. AWS warns against a large warmup value on a small training sample, because the rate might never reach that value before the run ends. On a few hundred examples that risk is live, and the result on the loss curve looks exactly like underfitting.
Batch size. Batch size is how many examples the job processes before it updates the model once. Larger batches average over more examples per update, which steadies each step and improves throughput, at the cost of memory. Smaller batches update more often on noisier estimates, which makes training less steady. On Bedrock it is batchSize, and whether there is anything to adjust depends entirely on the base model. The Nova understanding models do not expose it at all. Meta Llama 3.x and Titan Text Premier pin it at 1. Anthropic Claude 3 models allow 4 to 256 with a default of 32. Reach for it after epochs and learning rate, as a stability and throughput adjustment rather than the main lever on quality.
The loss curves as the reading instrument. These are not a hyperparameter you set; they are the output you steer by. Training loss falling means the model is fitting the data it sees. Validation loss, computed on held-out data, tells you whether that fit generalises. Both falling is healthy. Validation flattening while training keeps falling is the onset of overfitting. Validation rising while training still falls is overfitting in progress. There is a precondition: validationDataConfig is optional on the job, and validation metrics come back only for jobs that were given a validation dataset. Skip it and you have one curve, which is the curve that cannot tell you when to stop.
The curves are files, not a dashboard. Bedrock writes them under the S3 prefix you gave the job as its outputDataConfig, in a folder named for the training job. training_artifacts/step_wise_training_metrics.csv carries a row per step with step_number, epoch_number, training_loss and perplexity. validation_artifacts/post_fine_tuning_validation/validation_metrics.csv carries the same columns, with validation_loss in place of training_loss. Plotting one against the other is how the run gets read. Summary figures also come back in the trainingMetrics and validationMetrics fields of a GetModelCustomizationJob or GetCustomModel response, but only the per-step CSVs show the shape.
Early stopping. Rather than committing to a fixed epoch count, early stopping monitors validation loss and halts the run once it stops improving. Where the base model supports it, the job takes two more hyperparameters. earlyStoppingThreshold is the minimum improvement in validation loss that counts as improvement, and earlyStoppingPatience is the tolerance for stagnation before the job stops. Anthropic Claude 3 models default those to 0.001 and 2; Cohere Command defaults them to 0.01 and 6. Cohere also takes evalPercentage, which carves a validation split out of the training data when no separate set was supplied. One caution on what this does: early stopping ends training near the turn, so the run does not continue into the memorising phase. It does not reach back and return an earlier checkpoint for you.
Evaluation
Side by side
| Symptom | Epochs | Learning rate | Batch size | Loss-curve tell | Correction |
|---|---|---|---|---|---|
| Model unchanged from base | Too few | Too low | ✗ | Both curves stay high, barely fall | More epochs, or a higher rate |
| Learns but thrashes | ✗ | Too high | Too small | Loss jumps around, no smooth descent | Lower the rate, raise batch size |
| Reproduces training replies | Too many | Too high | ✗ | Training falls, validation flattens then rises | Fewer epochs, or early stopping |
| Lost general capability | Too many | Too high | ✗ | Good training loss, poor held-out eval | Fewer epochs, lower rate, re-evaluate |
| Learning too slowly | Too few | Too low | ✗ | Both curves fall, never reach a floor | Higher rate, or more epochs |
| Healthy run | Right for the data | Stable | Steady | Both fall together, level off | Stop at the validation low |
The solution
Start from the base model’s default hyperparameters, which are set as a reasonable centre for that model, and change one thing at a time from what the curves show. The first run above is textbook underfitting: the style never set, so the correction is more passes, and a modestly higher learning rate if it still will not move. The second run is the opposite failure. Note what it is not evidence of. It does not show that a few hundred examples cannot take more epochs, because AWS guidance runs the other way on that. It shows that this run went past its own validation low, and the record of where that happened is sitting in the validation CSV.
So read the epoch count against the validation curve rather than against the row count. Find the step where validation loss bottomed out, convert it to an epoch number from the epoch_number column, and set epochCount at or just below it on the next run. That is a measurement rather than a guess. If the behaviour still will not set at the model’s maximum epoch count, the remaining levers are the learning rate, the warmup steps, and better training data. There are no further passes to add.
Treat the learning rate as the stability control. If the loss will not descend smoothly and jumps around between steps, the rate is too high; bring it down and the descent steadies. If the model learns cleanly but too gradually to arrive within the epoch range, the rate is too low; nudge it up, with the caveat AWS attaches, that faster convergence this way can bring instability with it. On base models exposing a multiplier rather than a raw rate, the same logic holds. Check the warmup steps too, because a long warmup on a few hundred examples can leave the rate short of its target for the whole run.
Leave batch size until epochs and rate are roughly right, then use it to smooth or speed the run, if the base model exposes it at all. A larger batch gives steadier updates and better throughput, and is the natural response to a noisy training curve once the learning rate has been checked. A smaller batch updates more often and can move a stalled run along. It is a supporting adjustment, and it is rarely where a bad custom model went wrong.
Whatever the loss curves show, the model that ships is chosen by a held-out evaluation rather than by the loss number. Run the custom model and the base model against the evaluation set you kept back, on the task you care about, and compare. The loss curve tells you when the run was healthy. The evaluation tells you whether the result beats what you started with, and whether it kept the general ability you needed. A model with a low training loss that loses to the base on held-out data is not a good model, and only the evaluation surfaces that.
Worked example
The dataset is three hundred prompt-and-completion pairs of house-style support replies, with sixty more held back as a validation and evaluation set. The base model allows 1 to 10 epochs and defaults to 2, and validationDataConfig points at the held-out file so the job emits a second curve.
The first job runs at the default of 2 and comes out sounding like the base. Training and validation loss both fell a little, then flattened high. That is underfitting: two passes over three hundred examples did not converge. The correction is to raise the epoch count and rerun, watching the curves rather than the output alone.
The next job runs at 10, the model’s maximum. Training loss sinks to a low floor. Validation loss falls, bottoms out partway through, then climbs for the back half of the run. The model at the final epoch reproduces training replies verbatim and has started failing on ordinary requests. Reading validation_metrics.csv, the low sits at step 412, which the epoch_number column puts in epoch 5. The learning rate stays at the default, because the descent was smooth with no thrashing, so the problem was passes rather than step size.
The third job sets epochCount to 5. An alternative is to keep the count at 10 and set earlyStoppingThreshold and earlyStoppingPatience, so the job halts when validation loss stops improving rather than running the schedule out. Either way the run ends near the turn instead of past it.
The final check is not the loss at all. The chosen custom model and the base model both run against the sixty held-out pairs. The custom model matches the house style, still handles the general requests the base did, and wins the comparison. That is the evidence the model is ready, and the loss curve alone could not have given it, because a lower training loss and a better model are not the same claim. The habit underneath this is the same one behind matching a prompt technique to the task shape: a setting that helped one job is not a universal good.
What’s worth remembering
- The two failure modes are opposite: underfitting performs poorly on the training set, and overfitting performs well on training data and poorly on validation data.
- AWS guidance is that larger datasets converge in fewer epochs and smaller datasets need more, which is the reverse of the common assumption; read the stopping point off the validation curve instead of the row count.
- Watch training and validation loss together, and supply a validation dataset, because a job without one emits only the curve that cannot tell you when to stop.
- Early stopping halts the run when validation loss stops improving, so training does not continue past the turn; it does not return an earlier checkpoint for you.
- Ranges are per base model and narrow:
epochCounttops out at 5 on Nova and 10 on Claude 3 and Llama 3.x, andbatchSizeis pinned or absent on several families. - The model that ships is chosen by a held-out evaluation against the base model, not by the loss number.