Cheat Sheet: Model Customisation

A dense pass over how you push a foundation model closer to your task, from cheapest to heaviest, and how it gets served once you have.

The ladder at a glance

A finished artefact has four homes, and which one it lands in is settled before training rather than after. Open or externally trained weights go to Bedrock Custom Model Import. A Bedrock-native fine-tune serves as its base model allows, which for several bases means sitting behind Provisioned Throughput. A model whose runtime you want to own goes to a SageMaker AI endpoint, deployed from an approved version in the SageMaker Model Registry. LoRALoRAA fine-tuning technique that trains a small low-rank matrix on top of the frozen base model, instead of updating every parameter. adapters and the other parameter-efficient adaptation techniques get hot-loaded onto a shared base, so several variants answer from one endpoint with one copy of the base in GPU memory.

Decision rules

• If the answer needs current or private facts, use RAG, not fine-tuning.
• If the facts are already in the weights and only the format or tone comes out wrong, fine-tune.
• If a whole domain’s vocabulary and concepts are missing, use continued pre-trainingContinued pre-trainingFurther training a base model on a pile of your unlabelled domain text to teach it vocabulary and style, rather than task behaviour., which now runs as a Nova recipe on SageMaker AI training jobs or HyperPod.
• If you want both fresh facts and consistent format, fine-tune for the format and add RAG for the facts.
• If a prompt tweak or a few-shot example fixes it, stop there.
• If inference cost or latency is the problem and quality is close enough, distil to a smaller student.
• If you have trained weights elsewhere and want Bedrock serving, use Custom Model Import. Provisioned Throughput does not cover imported models; its eligibility list is AWS-provided base models and Bedrock customisations of those. An imported model serves on demand and bills per Custom Model Unit instead.
• The serving surfaces open to a model follow from where its weights came from, and each meters something different: tokens consumed, reserved unit-hours, active minutes, or instance-hours. Matching a pricing tier to the rhythm of the traffic covers how those meters land on a bill.
• Serving a Bedrock-native fine-tune depends on which model you customised, not on the fact of customising: a fine-tuned Nova Micro, Lite or Pro and a fine-tuned Llama 3.3 70B deploy for on-demand inference and bill per token, while a fine-tuned Llama 3.1 8B has no on-demand path and needs Provisioned Throughput, priced on the base model’s units.
• If you have only a few hundred clean examples, fine-tune; do not reach for continued pre-training.
• If your data is unlabelled bulk text, that is continued pre-training, not fine-tuning.
• If validation lossLoss curveThe plot of training error over time; the gap between the training and validation lines is how you spot memorising rather than learning. rises while training loss falls, you are overfitting; cut epochsEpochOne complete pass over the training dataset – more passes means more chance to shift behaviour, and more chance to memorise. or add data.
• If both losses stay high, you are underfitting; raise epochs or the learning rateLearning rateHow far each training step moves the model’s weights – too low and nothing shifts, too high and it lurches past what you wanted..
• If response quality can be scored by code or by a judge model, use reinforcement fine-tuning and put the scoring in the reward function. Preference-pair methods such as DPO run in the Nova recipes on SageMaker AI.
• If you cannot measure whether customisation helped, build a held-out set before you train.
• The two version stores cover different things. SageMaker Model Registry versions a model package: the artefact, the inference container, the evaluation metrics, the model card, and the approval status an automated deployment pipeline reads before it promotes anything. Bedrock versions the things you call: model ids, saved prompts and guardrails each carry a version, and each has to be pinned by whatever calls it.
• Rollback follows the serving surface. A SageMaker AI endpoint rolls back through deployment guardrails, shifting traffic in a canary or linear pattern with auto-rollback alarms watching the new fleet, so a failed deployment reverts without a human in the loop. A Bedrock custom model has no traffic shifting to configure: it rolls back by repointing the model id in the application, or by pointing the Provisioned Throughput commitment at the previous custom model.
• Retiring a model is three moves, not one. Reject the package version so no pipeline can promote it again, release the endpoint or the Provisioned Throughput commitment so it stops billing, and keep the artefact and the eval set it was scored on so a past result can still be reproduced.

Traps

• Fine-tuning does not teach new facts. It shapes behaviour. New facts come from RAG or continued pre-training.
• Continued pre-training needs large unlabelled corpora; fine-tuning takes smaller labelled prompt-completion pairs. Swapping them is the usual mix-up.
• Where Provisioned Throughput is forced, it is a standing cost that bills whether or not traffic arrives, which is why the model you start from sets the shape of the bill as much as the training does.
• More data is not automatically better. A smaller, clean, deduplicated set beats a large noisy one.
• Leaving validation examples in the training split leaks the labels and inflates your metrics.
• Skipping a train/validation split means you cannot see overfitting at all.
• PII and duplicates left in the dataset degrade the model and create compliance exposure.
• Distillation needs a teacher to label the data; the student is trained on the teacher’s outputs, not raw ground truth.
• Custom Model Import is for bringing weights in, not for training. It does not fine-tune anything, and an imported model cannot be used with batch inference.
• Continued pre-training is no longer one of Bedrock’s own customisation jobs. Bedrock runs supervised fine-tuning, reinforcement fine-tuning and distillation; training on an unlabelled corpus moved to the Nova recipes on SageMaker AI.
• Evaluating only against your fine-tuned model tells you nothing; compare against the base model on the same held-out set.
• Raising epochs endlessly does not keep improving quality; past a point it overfits.
• Reinforcement fine-tuning and supervised fine-tuning are different mechanisms. One grades sampled responses with a reward function you write, the other trains on labelled prompt-completion pairs.
• An adapter is worthless without the exact base model version it was trained against, so the base version travels in the release bundle alongside the adapter. Swap the base underneath a hot-loaded adapter and behaviour shifts with nothing in the adapter to explain it.
• A Provisioned Throughput commitment term turns a rollback into a cost decision. Reverting to the previous custom model mid-term does not release the units you committed to, so either you repoint the commitment at the older model or you carry two.

Say it in one line

1. Prompt engineering and RAG change the input, not the weights; fine-tuning and continued pre-training change the weights.
2. RAG is the move for fresh or proprietary facts.
3. Fine-tuning is the move for behaviour, format, and tone.
4. Continued pre-training is the move for domain knowledge and vocabulary, and it runs on SageMaker AI, not in Bedrock.
5. Fine-tuning datasets are labelled JSONL prompt-completion pairs.
6. Continued pre-training datasets are large volumes of unlabelled text.
7. Quality and cleanliness of data beat sheer volume for fine-tuning.
8. Always split train and validation, and strip PII, duplicates, and leakage first.
9. The hyperparametersHyperparameterA training setting you choose before the run (epochs, learning rate, batch size), as opposed to a weight the run learns. you set are epochs and learning rate on every base, plus batch size or learning-rate warmup steps depending on the model.
10. Watch validation loss: rising while training loss falls means overfitting; cut epochs.
11. Which serving paths a custom model can use is a property of the model it was built from: some serve on demand per token, some force Provisioned Throughput, and imported weights bill per Custom Model Unit per minute.
12. Custom Model Import brings open or custom weights into Bedrock managed serving.
13. Distillation produces a smaller, cheaper student from a teacher model.
14. Evaluate the custom model against a held-out set and against the base model before you trust it.