Lab 12 — Fine-tune a model and read the loss curves

Scaffold: 2/5. The datasets, the job launch, and the download are written. You write the part that turns two CSVs into a decision.

Cost, up front. Part A runs a real Amazon Bedrock customization job. On Nova Micro with the 300-record dataset here, the training charge is small (billed per token processed, multiplied by the epoch count) but it is not zero, and the resulting custom model is billed for storage every month until you delete it. The job runs for hours, not minutes. Part B stands the model up for inference: on-demand deployment costs per token and is cheap, one Provisioned Throughput model unit costs tens of US dollars an hour from creation to deletion. Rates move; check the Amazon Bedrock pricing page before you run anything.

There is a free path. ./scripts/test.sh runs the whole curve-reading exercise against two shipped sample runs, with no AWS account and no spend. If you only want the skill, that is the whole lab.

The scenario

A support team has a few hundred prompt-and-completion pairs that capture how their replies should sound, and a smaller set they kept back. They want a model that answers in the house style without being reminded of it in every prompt, so they can stop paying for a long few-shot preamble on every call.

The first run used the default hyperparameters and produced a model that sounded like the base. The second run cranked the passes right up and produced a model that parrots the training replies. Somewhere in between is the model they want, and nobody wants to find it by launching twenty jobs and reading the output.

The knobs are epochs, learning rate, and (on some base models) batch size. The signal is the pair of loss curves the job writes to S3 when it finishes. This lab is the mechanics of both.

The requirement

Launch a customization job with hyperparameters you chose, fetch the two metrics files it wrote, and produce a chart plus a one-paragraph reading that names which of three pictures the run is: underfitting, healthy, or overfitting. On an overfitting run, name the step where the run should have stopped.

What’s provided

• template.yaml — an S3 bucket for the datasets and the job output, plus the IAM service role Bedrock assumes to read one and write the other. There is no CloudFormation resource for a customization job, so the job itself is launched by script.
• data.py — generates training.jsonl (300 pairs), validation.jsonl (60), and heldout.jsonl (40), all disjoint, in the Bedrock conversational fine-tuning format ("schemaVersion": "bedrock-conversation-2024").
• src/plot_curves.py — file discovery, CSV parsing, coordinate maths, and the stylesheet. The gaps are render_svg() and verdict().
• solution/plot_curves.py — the reference answer.
• sample-output/healthy/ and sample-output/overfit/ — two complete sets of real-shaped metrics CSVs, laid out exactly as Bedrock writes them, so the reading exercise works with zero AWS spend.
• scripts/ — deploy, train, fetch-metrics, test, serve-and-compare, teardown.

The house style

Every reply in the dataset does the same four things: opens with the subscriber’s first name, states the fact in one plain sentence, gives the next action, and closes with “Any trouble, just reply to this email.” Every detail a reply repeats appears in the message it is replying to, so the model is learning a shape rather than learning to invent facts. A style this mechanical is one you can grade by eye, which is what makes the Part B comparison readable without a judge model.

Your task

Implement two functions in src/plot_curves.py:

1. render_svg(training, validation, title) — return an SVG string plotting both curves against step number, with a marker at the lowest validation point. _bounds(), _points() and _STYLE are written; this is assembly.
2. verdict(training, validation) — return one short paragraph naming the picture:
   • underfitting: training loss barely moved from where it started;
   • overfitting: validation loss bottomed out early and climbed after, while training loss kept falling;
   • healthy: both fell together and validation has not turned up.

   On an overfitting run, say the step the validation low sits at. That is the best model the run produced, and where the epoch count should have ended.

Training loss alone always looks like progress, because a model can always fit its own training data harder. The validation curve is the one that tells you when that progress stopped being real, and verdict() is where you encode that.

Run it

The free path (no AWS account, no spend)

./scripts/test.sh              # runs your src/plot_curves.py
SRC=solution ./scripts/test.sh # runs the reference answer

Writes build/healthy.svg and build/overfit.svg from the shipped samples and prints a verdict for each.

Part A: train a model and fetch its curves (paid, hours)

./scripts/deploy.sh        # bucket, role, datasets uploaded. Cents.
./scripts/train.sh         # launches the job. Confirms first. Hours.
./scripts/fetch-metrics.sh # downloads the CSVs and plots them

Hyperparameters are environment variables, so you can rerun with a different shape:

EPOCH_COUNT=5 LEARNING_RATE=0.00005 ./scripts/train.sh

Amazon Nova Understanding models expose exactly three, with the defaults and ranges from the Bedrock user guide’s Custom model hyperparameters:

There is no batchSize and no learningRateMultiplier on Nova. Those exist on other base model families (Anthropic Claude 3 exposes batchSize, learningRateMultiplier, earlyStoppingThreshold and earlyStoppingPatience; Cohere Command exposes batchSize and early stopping; Meta Llama exposes batchSize fixed at 1). The set of knobs is a property of the base model, not of fine-tuning, which is worth knowing before you go looking for a dial that is not there. Warmup is recommended at roughly dataset size divided by 640 for Nova Micro, so 300 records wants a very small number; the default is 2 here.

Part B: serve it and compare (optional, paid, gated)

./scripts/serve-and-compare.sh              # on-demand deployment, per token
MODE=provisioned ./scripts/serve-and-compare.sh  # 1 no-commitment PT model unit

It prints what it will cost, refuses to move until you type a confirmation, and deletes the serving capacity on exit, on Ctrl-C, and on error. It runs held-out messages through the base model and the custom model side by side, which is the comparison that decides whether the model ships. The loss curve says the run was healthy; only this says the model is better.

Teardown

./scripts/teardown.sh

Deletes deployments and Provisioned Throughputs first (they meter), then the custom model, then the bucket contents and the stack. The custom model is not a CloudFormation resource, so deleting the stack does not touch it; it goes through aws bedrock delete-custom-model --model-identifier <name-or-arn>, and teardown does that for you unless you pass KEEP_MODEL=1.

What success looks like

./scripts/test.sh writes two SVGs and prints two readings. The healthy run should read as healthy: both curves fall together and level off. The overfitting run should name its turning point, somewhere around step 320 in epoch 3, and say the epoch count should have ended near there. Open the two SVGs side by side; the shape difference is the whole lesson.

On Part A, fetch-metrics.sh should find both CSVs under the output prefix and plot your own run. Every job writes to the same output/ prefix under its own model-customization-job-<id>/ folder, so once you have trained twice there are two sets there. The script plots the job in build/job-arn.txt, the one you launched last, and prints which id it used; set JOB_ID=<id> to plot an earlier run instead.

If it fails

• fetch-metrics.sh finds nothing — the CSVs are not written until the job reaches Completed. Check with aws bedrock get-model-customization-job --job-identifier "$(cat build/job-arn.txt)" --query status.
• The poll stopped before the job finished — Ctrl-C, a dropped connection, or credentials expiring mid-run. train.sh retries the status call a few times before it gives up, and the job itself is unaffected either way: it keeps running, and re-running train.sh would launch a second paid one. Wait for Completed, then write the file Part B reads: aws bedrock get-model-customization-job --job-identifier "$(cat build/job-arn.txt)" --query outputModelArn --output text > build/model-arn.txt.
• No validation_metrics.csv — you launched without a validationDataConfig. Without it you get one curve and no way to tell learning from memorising. Relaunch with the validation set.
• ValidationException on the hyperparameters — the values in the hyperParameters map must be strings, not numbers, and every key has a base-model-specific range. batchSize on a Nova base model is rejected because Nova does not have one.
• AccessDeniedException when creating the job — your own identity needs iam:PassRole for the service role, on top of bedrock:CreateModelCustomizationJob.
• The job fails on the data — every record must be one line of valid JSON in the format the base model expects. Nova wants the conversational shape; {"prompt": ..., "completion": ...} is a different format for a different family. Check with head -1 build/training.jsonl | python3 -m json.tool.
• Part B create-custom-model-deployment fails — on-demand custom model deployment is limited by base model and Region (Nova in us-east-1, Llama 3.3 70B in us-west-2). Use MODE=provisioned if your model is not eligible, and delete it the same hour.
• The custom model still shows in the console after teardown — that is storage you are paying for. aws bedrock list-custom-models and delete it.

Reveal the solution

SRC=solution ./scripts/test.sh

What you just learned

• Two curves, not one. Training loss measures fit on the data the model learns from; validation loss measures fit on held-out data it does not. Training loss always falls, because a model can always fit its own training data harder. The gap between them is the signal.
• The turning point is the model you wanted. Where validation loss bottoms out and starts to climb is where the run stopped generalising and started memorising. Everything after that point is a worse model with a better training loss.
• Epoch count scales with dataset size. A few hundred examples turn from learning to memorising after a couple of passes. Cranking epochs to force a stubborn behaviour on a small dataset is how you get a parrot.
• The knobs belong to the base model. epochCount is nearly universal; batchSize, learningRateMultiplier and early stopping exist on some families and not others. Read the hyperparameter table for the model you picked before you plan a sweep.
• validationDataConfig is what buys you the second curve. Skip it and the job still succeeds, still reports a training loss, and tells you nothing about whether the model generalises.
• Serving a custom model is a separate cost decision. On-demand custom model deployment bills per token; Provisioned Throughput bills by the hour from creation to deletion whether you use it or not. The forgotten Provisioned Throughput is the expensive mistake in this whole track.
• The loss curve is not the ship decision. It tells you the run was healthy. A held-out comparison against the base model tells you the result is better, and those are different claims.

Next

Lab 13 — Generate the week’s creative from the box manifest. Fine-tuning taught the model to sound like Greenbox; the next lab teaches the pipeline to draw like it, wiring Stable Image Core and Luma Ray 2 onto the weekly box manifest.