Combining RAG and Fine-Tuning for a Legal Contract Assistant

The situation

A legal-technology startup is building a contract review assistant for a mid-sized commercial firm. The in-product model answers two shapes of question: “What does this clause mean in the context of our past drafting?” and “Where have we seen this indemnity construction before, and how did we negotiate it?”

The constraints:

• Corpus: ~200,000 past contracts, amendments, side letters, and internal case studies. Roughly 40 GB of text-heavy PDFs, Word documents, and Markdown notes after extraction. Growing by ~500 new matters a month.
• Voice: every answer references clauses by section number (§3.2(b)), uses the firm’s preferred hedging (“the drafting is ambiguous on this point” rather than “this is unclear”), and cites internal precedents in the firm’s matter-number format.
• Refusal: questions outside commercial contract law (tax, immigration, employment) get a structured decline with a pointer to the correct in-house team. Nothing off-domain.
• Budget: AUD$100,000 end-to-end for customisation, data preparation, trainingTrainingThe process of fitting a model’s weights to data by minimising a loss function., evaluation, first quarter of inferenceInferenceRunning a trained model to produce output – as opposed to training it..
• Timeline: three months to a pilot with fee-earners.
• Platform: Bedrock. Nothing self-hosted.

What actually matters

Start with what kind of problem “be correct about 200,000 contracts” is. It’s a retrieval problem. Facts about specific documents live in the documents, and a model trained to memorise 200,000 of them is either ruinously expensive to train or wrong in ways nobody can trace back to a source. That pushes the “what does the corpus say?” half of the design toward retrieval, which makes the vector storeVectorAn ordered list of numbers – in AI usage, almost always an embedding – and by extension the databases that index them for nearest-neighbour search. and the embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. model the interesting choices.

The other half is a behaviour problem. The firm’s voice is a set of rules: hedged phrasings, §-citations, matter-number formats, a structured decline when the question drifts into tax law. Rules about how to write are patterns of output conditioned on input rather than facts about the world. A system promptSystem promptThe instruction block that frames the model’s behaviour for a session, separate from the user’s messages. carries them up to a point, and then output drifts under adversarial phrasing and over long conversations. Supervised fine-tuning moves the rules into the weights, so a short prompt is enough and the behaviour survives a prompt engineered to strip a style-guide line out. The labelled dataset becomes the artefact that encodes the style guide.

Then the refresh cadence. The corpus grows by 500 matters a month. The style guide changes when a senior partner wins an argument about hedging. The refusal list changes when a user finds a new way to ask about divorce. A two-person platform team can absorb weekly ingestion on object-storage events and quarterly fine-tune refreshes, and cannot absorb monthly retrains over 40 GB. The third lever settles itself anyway. Bedrock’s customisation methods are supervised fine-tuning, reinforcement fine-tuning and distillation; continued pre-training is not among them any more, so domain adaptation on raw text means training outside Bedrock and importing the weights.

Where the money goes decides the rest. AUD$100K over three months looks like training compute and turns out to be serving. A full-rank fine-tuned model runs only on provisioned throughput, at a fixed hourly rate from the day it deploys, and that line is the one most often under-estimated. Retrieval has the opposite shape: near-zero fixed cost, variable per query. Evaluation splits the same way. A model that gets the voice right and invents clause numbers is worse than an untuned one that cites accurately, so citation scores and voice scores have to move independently, or the team can’t tell which half to fix.

What we’ll filter on

Five filters to score the landscape against.

1. Corpus groundingGroundingConstraining a model to answer from provided sources rather than from whatever it absorbed during training.. Two hundred thousand documents the model has never seen, with new ones arriving weekly. Answers have to reflect the current corpus, not a snapshot frozen at training time.
2. Voice and format. The firm’s phrasing and citation style are rules about how to write, not facts about the world. They need to hold without a prompt re-teaching them every turn.
3. Refusal. Off-domain questions must be declined in a structured way, and the policy has to hold under adversarial prompting.
4. Budget and timeline. AUD$100K and 90 days. Any method that blows either is out.
5. Maintainability. A two-person platform team. Customisation has to be refreshable when the corpus grows or the style guide changes, without a full retrain every time.

The landscape

Bedrock offers four levers that could shape behaviour here, and one it used to.

Prompt engineering alone. Cheapest. System prompt with the style guide, few-shot examples, refusal instructions. Works well for voice and refusal when the base model is capable; Claude Sonnet follows detailed style instructions closely. Fails the corpus filter, because 200,000 documents don’t fit in any prompt.

Retrieval-augmented generation. The corpus lives in a vector store; every question retrieves relevant chunks, and those chunks ride into the prompt alongside the user’s question. Facts stay outside the weights, so updating the corpus is an ingestion job rather than a training job, and every claim traces back to the chunk it came from. On Bedrock: Knowledge Bases plus RetrieveAndGenerate, over OpenSearch Serverless, OpenSearch managed clusters, S3 Vectors, Aurora, Neptune Analytics, Pinecone or MongoDB Atlas.

Supervised fine-tuning. Show a base model a labelled dataset of (prompt, ideal response) pairs; adjust weights so outputs move closer to the ideal. Bedrock fine-tunes Amazon Nova Micro, Lite, Pro and Nova 2 Lite in us-east-1, and Meta Llama 3.1 8B and 70B, Llama 3.2 1B, 3B, 11B and 90B, and Llama 3.3 70B in us-west-2. Not the Llama 4 models. Claude 3 Haiku was the one Anthropic model on that list and reached end of life on 10 September 2026, so no Anthropic model can be fine-tuned now, and the Titan text models have left the catalogue entirely. Training itself is cheap: Nova Lite is USD$0.002 per 1,000 training tokensTokenThe unit of text an LLM actually sees – usually a short character sequence, not a whole word., Nova 2 Lite USD$0.00378, and custom model storage is USD$1.95 a month. Teaches style, format and behaviour; does not reliably teach facts.

Continued pre-training. Keep training a base model on a large body of unlabelled domain text using the objective that originally pre-trained it, shifting its distribution of language toward the domain. Bedrock ran this on Amazon Titan Text, those models are gone, and the user guide no longer lists it among the customisation methods. The CONTINUED_PRE_TRAINING enum survives in the API with no current base model behind it. Worth understanding as a technique, and not a choice available here.

Bedrock Custom Model Import. Bring weights trained elsewhere (Llama, Mistral, Mixtral, Flan-T5, Qwen or GPT-OSS architectures) and serve them through the Bedrock API. Billed per Custom Model Unit per minute in five-minute windows, scaling to zero after five idle minutes with a cold start of tens of seconds; available in us-east-1, us-east-2, us-west-2 and eu-central-1. This is now the route for domain-adapted weights, and it’s a packaging choice rather than a fresh customisation lever.

Evaluation

Side by side

Lever	Corpus grounding	Voice & format	Refusal behaviour	Budget/timeline	Maintainability
Prompt engineering alone	✗	✗	✗	✓	✓
RAG (Knowledge Bases)	✓	✗	✗	✓	✓
Supervised fine-tuning	✗	✓	✓	✓	✓
Custom Model Import	✗	✓	✓	✗	✗

Continued pre-training isn’t in the table because it isn’t on the menu. No single remaining lever clears all five filters. Two stacked clear all five: RAG for the corpus, fine-tuning for voice and refusal.

Matching the levers to the question

The solution

Knowledge Bases over the 200,000 documents: Titan Text Embeddings V2 at 1,024 dimensions, hierarchical chunking, metadata filters on matter number and practice area, weekly incremental ingestion from S3 via EventBridge calling StartIngestionJob. Supervised fine-tuning of Amazon Nova Lite on ~1,500 lawyer-curated pairs covering voice, §-citation format and structured refusals, then deployed for on-demand inference and called as the generator behind RetrieveAndGenerate.

Three mechanics decide whether that assembles.

The custom model has to be deployable on demand. Bedrock’s on-demand custom deployment covers Nova Micro, Lite, Pro and Nova 2 Lite in us-east-1, and Llama 3.3 70B in us-west-2, and the model must have been customised on or after 16 July 2025. Create the deployment with CreateCustomModelDeployment and pass its ARN as modelId. Anything else customised on Bedrock, including any full-rank fine-tune, runs on provisioned throughput instead: a custom Llama 3.1 70B is USD$24 an hour per model unit with no commitment, around USD$17,000 a month whether or not anyone asks it a question.

Region pins the rest of the design. Nova fine-tuning runs only in us-east-1 and on-demand deployment of a custom Nova only in us-east-1, so the knowledge base and the vector store belong there too.

A knowledge base pointed at a custom model needs its orchestration and generation prompt templates supplied explicitly, with the information variables for the user’s input and the retrieved context. A base model gets defaults; a custom one doesn’t. Keep the $output_format_instructions$ placeholder in the orchestration template while rewriting, because without it the response comes back with no citations and no error.

Two smaller limits are worth knowing before committing. Files cap at 50 MB each and one ingestion job takes up to 5,000,000 new or updated files, so 200,000 contracts and 500 a month are nowhere near the edge. Nova Lite caps output at 5,000 tokens, which is comfortable for a clause answer and tight for a full memo.

Worked example

Attribute 1, the 200,000-document corpus. Knowledge Bases ingests into OpenSearch Serverless. Titan Text Embeddings V2 at 1,024 dimensions, one of the three widths it supports alongside 256 and 512. Hierarchical chunking, child ~300 tokens for retrieval precision, parent ~1,500 tokens for generator context. Metadata sidecars tag each document with matter number, practice area and client. Weekly refresh on deltas only. The fine-tuned model calls the same vector store an untuned one would.

Attribute 2, voice and citation format. A lawyer-in-the-loop curates ~1,500 (prompt, ideal-response) pairs over four to six weeks. Each pair is a real exchange, reviewed and edited to the style guide: hedged phrasing, §X.Y(z) references, matter-number citations. That dataset trains Nova Lite. Llama 3.3 70B is the alternative if quality demands it, and it’s the only Meta model that also deploys on demand.

Attribute 3, refusal on off-domain questions. A subset, perhaps 300 of the 1,500 pairs, are refusal examples, which puts the behaviour in the weights. The system prompt reinforces it, and the default holds better under a prompt-injection attempt than a prompt-only approach does.

Attribute 4, AUD$100K and 90 days. Budget pass below; both methods fit.

Attribute 5, maintainability. RAG updates are ingestion, with no retrain when a new matter lands. Fine-tuning refreshes quarterly, when the style guide evolves or refusal patterns grow. A two-person team runs ingestion continuously and the fine-tune four times a year.

Cost shape: where the dollars land

The cost profile differs in shape, not just size.

RAG: near-zero fixed, variable with queries. Embedding the whole 40 GB once at Titan Text Embeddings V2’s USD$0.02 per million tokens is about USD$200 for roughly 10 billion tokens, and the weekly deltas are noise next to that. OpenSearch Serverless bills USD$0.24 per OCU-hour, minimum capacity is set per collection group and configurable down to zero, and capacity steps in units of two OCUs when it runs.

Fine-tuning: low training, with serving cost depending on the family. A few million training tokens at Nova Lite’s USD$0.002 per 1,000 is low tens of US dollars, plus USD$1.95 a month of model storage. Serving is the larger number. A custom Nova deployed on demand is charged at the base model’s token rates, USD$0.06 per million input and USD$0.24 per million output for Nova Lite, so inference scales with usage rather than with the calendar.

Budget pass, AUD:

• Data preparation. PDF extraction, chunking pipeline, metadata tagging, the 1,500-pair dataset curated by a lawyer: ~AUD$30K.
• Embedding the corpus plus OpenSearch Serverless capacity for three months: ~AUD$5K.
• Fine-tune training plus iteration cycles: ~AUD$1K.
• Weekly Bedrock evaluation runs against a 200-question golden set: ~AUD$4K.
• Generation for the pilot at low query volume, on the fine-tuned Nova Lite deployment: ~AUD$2-4K.

Total: ~AUD$42-44K of AUD$100K. A Llama 3.1 70B fine-tune instead would add a provisioned-throughput line of roughly USD$52,000 over the three months, most of what’s left, for a pilot serving a handful of fee-earners. On-demand custom serving is what keeps that line out and leaves room for a Sonnet evaluation judge, more iteration cycles, and a Llama comparison run if quality wobbles.

Scoring the two halves separately

A contract review assistant that gets the voice right and invents clauses is worse than one that gets the voice roughly right and cites accurately. Evaluation carries as much weight as the customisation choice.

The golden dataset: ~200 real questions from the firm’s advice history, with expected answers reviewed by a senior lawyer. Refreshed quarterly. Includes questions the system should decline.

Bedrock’s retrieve-and-generate evaluation splits the signal for you. Builtin.CitationPrecision and Builtin.CitationCoverage measure whether § references trace back to retrieved chunks, Builtin.Faithfulness whether the answer stays inside them, and Builtin.Correctness and Builtin.Completeness score against the lawyer-reviewed reference. Refusal behaviour has no built-in metric and needs a custom one. Citation scores move when retrieval changes; refusal and voice scores move when the fine-tune drifts.

Human review: a weekly spot check by a senior lawyer on a random sample, scored on “would I have said it this way?” When rubric scores fall, the fine-tune dataset needs refreshing. When citation scores fall, retrieval is returning the wrong chunks.

What’s worth remembering

1. RAG and fine-tuning answer different questions, what the corpus says and how the answer should be written, so stacking them is the ordinary shape rather than a compromise. Facts live in the vector store; rules about phrasing live in the weights.
2. Bedrock’s customisation methods are supervised fine-tuning, reinforcement fine-tuning and distillation. Continued pre-training has no base model left behind it now that the Titan text models are gone, and domain-adapted weights arrive through Custom Model Import instead.
3. Fine-tuning covers Nova Micro, Lite, Pro and Nova 2 Lite in us-east-1, and Llama 3.1, 3.2 and 3.3 in us-west-2. Claude 3 Haiku reached end of life on 10 September 2026, so no Anthropic model is on that list, and neither are the Llama 4 models.
4. On-demand custom deployment covers the Nova family and Llama 3.3 70B, charged at base-model token rates, and needs a model customised on or after 16 July 2025. Everything else, full-rank fine-tunes included, runs on provisioned throughput at a fixed hourly rate from the day it deploys.
5. A knowledge base pointed at a custom model needs its orchestration and generation prompts supplied explicitly, and citations depend on the $output_format_instructions$ placeholder surviving that rewrite.