Cheat Sheet: Model Customization

A dense pass over how you push a foundation model closer to your task, from cheapest to heaviest, and how it gets served once you have.

The ladder at a glance

Decision rules

• If the answer needs current or private facts, use RAG, not fine-tuning.
• If the model knows the facts but replies in the wrong format or tone, fine-tune.
• If the model lacks a whole domain’s vocabulary and concepts, use continued pre-trainingContinued pre-trainingFurther training a base model on a pile of your unlabelled domain text to teach it vocabulary and style, rather than task behaviour. .
• If you want both fresh facts and consistent format, fine-tune for the format and add RAG for the facts.
• If a prompt tweak or a few-shot example fixes it, stop there; it is the cheapest rung.
• If inference cost or latency is the problem and quality is close enough, distil to a smaller student.
• If you have trained weights elsewhere and want Bedrock serving, use Custom Model Import.
• If you must serve a customised Bedrock model, buy Provisioned Throughput; on-demand does not serve custom models.
• If you have only a few hundred clean examples, fine-tune; do not reach for continued pre-training.
• If your data is unlabelled bulk text, that is continued pre-training, not fine-tuning.
• If validation lossLoss curveThe plot of training error over time; the gap between the training and validation lines is how you spot memorising rather than learning. rises while training loss falls, you are overfitting; cut epochsEpochOne complete pass over the training dataset – more passes means more chance to shift behaviour, and more chance to memorise. or add data.
• If both losses stay high, you are underfitting; raise epochs or the learning-rate multiplierLearning rateHow far each training step moves the model’s weights – too low and nothing shifts, too high and it lurches past what you wanted. .
• If you want the model to prefer certain response styles by human judgement, use RLHF or preference tuning.
• If you cannot measure whether customisation helped, build a held-out set before you train.

Traps

• Fine-tuning does not teach new facts. It shapes behaviour. New facts come from RAG or continued pre-training.
• Continued pre-training wants large unlabelled corpora; fine-tuning wants smaller labelled prompt-completion pairs. Swapping them is a classic distractor.
• A customised Bedrock model cannot be called on-demand. Provisioned Throughput is required to serve it, and that is a standing cost.
• More data is not automatically better. A smaller, clean, deduplicated set beats a large noisy one.
• Leaving validation examples in the training split leaks the answer and inflates your metrics.
• Skipping a train/validation split means you cannot see overfitting at all.
• PII and duplicates left in the dataset degrade the model and create compliance exposure.
• Distillation needs a teacher to label the data; the student is trained on the teacher’s outputs, not raw ground truth.
• Custom Model Import is for bringing weights in, not for training. It does not fine-tune anything.
• Evaluating only against your fine-tuned model tells you nothing; compare against the base model on the same held-out set.
• Raising epochs endlessly does not keep improving quality; past a point it overfits.
• RLHF and standard supervised fine-tuning are different mechanisms; preference data is ranked, not simple prompt-completion pairs.

Say it in one line

1. Prompt engineering and RAG change the input, not the weights; fine-tuning and continued pre-training change the weights.
2. RAG is the move for fresh or proprietary facts.
3. Fine-tuning is the move for behaviour, format, and tone.
4. Continued pre-training is the move for domain knowledge and vocabulary.
5. Fine-tuning datasets are labelled JSONL prompt-completion pairs.
6. Continued pre-training datasets are large volumes of unlabelled text.
7. Quality and cleanliness of data beat sheer volume for fine-tuning.
8. Always split train and validation, and strip PII, duplicates, and leakage first.
9. Key hyperparametersHyperparameterA training setting you choose before the run (epochs, learning rate, batch size), as opposed to a weight the run learns. are epochs, learning-rate multiplier, and batch size.
10. Watch validation loss: rising while training loss falls means overfitting; use early stopping.
11. A customised Bedrock model needs Provisioned Throughput to serve.
12. Custom Model Import brings open or custom weights into Bedrock managed serving.
13. Distillation produces a smaller, cheaper student from a teacher model.
14. Evaluate the custom model against a held-out set and against the base model before you trust it.