The situation
An online retailer runs a customer-support assistant on Amazon Bedrock. It handles around four thousand chat conversations a week: where is my delivery, can I return this, can you change the address on order 40118. Each call to the model carries a set of standing instructions, the conversation so far, and a few passages pulled out of the company’s help centre by a search over its articles and its returns policy.
Two complaints keep arriving from the support team, and they look like one problem until you read the transcripts side by side. In the first kind, the assistant states a refund rule that was retired in March, or tells a customer it cannot help with something the current policy plainly covers. In the second kind, the assistant is fine for five or six turns and then asks for an order number the customer already gave in turn two, or forgets that this is a gift order and quotes standard delivery timings back at them.
The team’s first instinct is to fine-tune: train the model on the current policies and the behaviour will follow. Nothing in the transcripts supports that yet. Both failures are about what reached the model on a particular call, not about what the model is capable of.
What actually matters
Start with the property that makes all of this necessary. A call to a foundation model is stateless. The model holds nothing from the previous message, the previous turn, or the previous conversation. If the model’s answer refers to something the customer said four turns ago, it is because the application sent those four turns along with the current question. There is no hidden memory to consult, only the text of this call.
That text has a ceiling. The context window is the maximum amount of material a single call may contain, counted in tokens, and it covers both what goes in and the room left for what comes back. A token is a piece of text a little shorter than a word. A thousand tokens is roughly 750 words of ordinary English, and rare words and identifiers split into several tokens each. So a model advertising an 8,000-token window has room for about 6,000 words of everything at once: the instructions, any examples, the retrieved help-centre passages, the transcript so far, anything a tool returned, and the reply.
Everything in that list competes for the same space. Adding three more retrieved articles means something else leaves. When the total goes over the ceiling, one of two things happens. Either the API rejects the call outright, which is loud and easy to find, or the application has already trimmed the prompt to fit, usually by dropping the oldest turns of the conversation, and that is silent. The second complaint from the support team is exactly that: history fell off the front of the prompt and nobody was told. The first complaint is the same shortage from the other side. Retrieval either found nothing relevant, or returned whole documents so large that one of them filled the window on its own.
Size costs money and time as well as space. Input tokens are billed on every call, and a longer prompt takes longer to process before the first word of the answer appears. An assistant that replays the whole transcript every turn gets steadily more expensive as the conversation runs, so the tenth turn can cost several times what the first one did for no gain in quality.
Deciding what occupies that budget on each call is context engineering. It is a level up from prompt engineering, which is about the wording of an instruction: how to phrase the task, whether to show examples, how to ask for a particular output shape. Context engineering asks which material is present on this call at all, in what form, and what gets left out to make room for it. Both apply to any application built on foundation models [FMs], and neither one touches the model itself.
What we’ll filter on
- Constant or per-request: is this the same on every call, or does it change with the customer’s question?
- Growth: does it stay a fixed size, or get bigger the longer the conversation runs?
- Fidelity: does the model need the exact words, or is a summary as useful?
- Freshness: how recently does it have to have been updated to be correct?
- Share of the window: how much of the budget does one instance of it consume?
- Failure mode: what goes wrong when it is missing, and is that visible or silent?
The landscape
Six kinds of material turn up in a support assistant’s context window, and a real prompt is a stack of several of them.
The system prompt
The standing instructions that go at the top of every call: who the assistant is, what tone it uses, what it must refuse, what shape the answer should take, and how to behave when the retrieved material does not answer the question. It is constant across every call, so it can be written once, reviewed, and version-controlled. It is also charged for on every call, which puts a real price on every sentence that is not doing work. Deciding what belongs in it, and what needs a stronger control than an instruction is work in its own right.
Few-shot examples
A handful of worked examples of input and desired output, placed in the prompt so the model can copy the pattern. Asking with no examples is zero-shot; adding two or three is few-shot. They work well where the wanted output is easier to show than to describe, such as a fixed refund-decision format, and they cost their full length on every call that carries them.
Retrieved passages
The help centre is far larger than any context window, so the application searches it per request and includes only the passages that match. That search runs over chunks rather than whole articles. Chunking splits each source document into passages of a few hundred tokens, usually with a small overlap so a sentence straddling a boundary survives in one of them. Those chunks, not the articles, are what gets indexed and retrieved. Chunk size sets both how precisely a search can land on the right paragraph and how much of the window each hit consumes. Most sources are too big to send whole, which is the ordinary case here rather than the awkward one.
The conversation so far, verbatim
Every turn of the chat, replayed in full. Exact, simple to build, and it grows without limit. By turn twenty it is the largest thing in the prompt, and the parts that matter least, the pleasantries from turn one, are the parts that have been paid for the most times.
Summarised history
A short running summary of the conversation, rewritten as it goes, in place of the older turns. It stays roughly the same size no matter how long the chat runs, and it loses exact wording, which matters when the exact wording was an order number or a quoted price. Where the line falls between the turns kept verbatim and the ones folded into a summary decides how much of that loss you take.
Tool results
When the assistant calls something, an order-lookup API or a delivery-tracking service, the result comes back into the context window as more text for the model to read before it answers. Raw responses are often far longer than the answer needs: a tracking payload can carry forty fields when the reply uses three. These also arrive mid-call, so a prompt that comfortably fitted on the way out can be over the ceiling by the time the tool has answered.
Evaluation
Side by side
| Layer | Same on every call | Fixed size | Needed word for word | Small share of the window | Silent when missing |
|---|---|---|---|---|---|
| System prompt | ✓ | ✓ | ✓ | ✓ | ✗ |
| Few-shot examples | ✓ | ✓ | ✓ | ✗ | ✗ |
| Retrieved passages | ✗ | ✓ | ✓ | ✗ | ✓ |
| Verbatim history | ✗ | ✗ | ✓ | ✗ | ✓ |
| Summarised history | ✗ | ✓ | ✗ | ✓ | ✓ |
| Tool results | ✗ | ✓ | ✗ | ✗ | ✗ |
Read the second and third columns together and the design falls out. Only one layer grows without limit, and it is also the layer the model needs least precisely once a few turns have passed. That combination is why verbatim history is the first thing to convert rather than the first thing to truncate.
The last column is the one to take seriously in production. A system prompt that fails to load produces visibly strange behaviour within minutes. Retrieval that returns nothing, or history that quietly lost its first six turns, produces an answer that reads perfectly well and happens to be wrong, which is the harder failure to notice and the one both support complaints describe.
Where the budget goes
The solution
Assemble the context in layers, sized deliberately, and count them before the call goes out.
The system prompt comes down to a few hundred tokens of instructions the model does not already follow on its own. Anything it reliably does anyway is paying rent on every call for nothing. Few-shot examples go in only where the output shape is genuinely hard to describe, two of them rather than six, and only on the calls that produce that shape.
Retrieval returns passages, not documents. With the help centre chunked into a few hundred tokens per passage and the top five hits included, the refund rule arrives beside the customer’s question at a cost of around 1,500 tokens instead of 4,200, and there is room for five sources instead of three. Carry each chunk’s article title with it so the model can say where an answer came from.
History splits in two. The last few turns stay verbatim, because a follow-up question usually depends on the exact wording of the turn before it. Everything older becomes a running summary, rewritten as the conversation goes. On top of that, pull the facts that must never be lost, the order number, the gift flag, the delivery address the customer corrected, out of the prose and pin them as a short structured block. A summary is allowed to blur an apology; it is not allowed to blur order 40118.
Tool results get filtered at the application, not at the model. Return the three fields the answer uses and drop the other thirty-seven.
Then make the budget visible. Reserve the output room explicitly, count the tokens of each layer before sending, and log those counts. An assistant that trims silently is an assistant that will lose the customer’s order number one day and tell nobody. If the layers do not fit, that is a decision to make in code, with a rule about what goes first, and not something to leave to whichever library happens to be holding the prompt.
Placement is worth a small amount of care. Models attend less reliably to material buried in the middle of a long prompt. Put the standing instructions and the customer’s actual question at the two ends, with the retrieved passages in between.
All of this is the cheap lever, and it is the one to pull before fine-tuning. Fine-tuning changes the model’s weights: it needs a labelled dataset, a training job, an evaluation, and a redeploy, and when the refund policy changes again next quarter the whole cycle repeats. Changing what goes into the context window is a configuration change that can be tested this afternoon and rolled back this evening. Fine-tune when the behaviour you want cannot be described in instructions or shown in examples. Neither of the two complaints here is that. The model was never told the current policy on one call and never shown turn two on the other.
Worked example
Take the refund complaint. The customer asks whether a bike helmet bought eight weeks ago can be returned. Today’s assembly searches the help centre, gets three long articles back, and the returns-policy article that carries the sixty-day rule for safety equipment ranks fourth, so it never reaches the model at all. The model answers from the general thirty-day rule in article one and refuses a refund the company would have honoured. After chunking, the sixty-day paragraph is its own passage rather than page four of a long article. It ranks second on a query about helmets and returns, and costs 300 tokens to include.
Now the forgotten order number. At turn nine the old prompt is 1,100 tokens over the ceiling, so the trimmer drops turns one to four, which is where the customer typed 40118 and mentioned it was a gift. The model asks for the order number again, because from where it sits the customer never gave one. In the new assembly, turns one to five have become a 250-token summary and the order number and gift flag are pinned in a structured block that the trimmer is not allowed to touch. The whole call is 3,500 tokens, so nothing is being trimmed anyway, and there is room for the conversation to run another twenty turns before that changes.
What’s worth remembering
- A model call carries no memory of its own, so everything the model considers is text the application placed in one context window measured in tokens.
- Those layers compete for a single budget, and when the budget overflows the call either fails loudly or gets trimmed silently, which is what a customer means when they say the assistant forgot what they told it.
- Context engineering decides which material is present on a call and in what form; prompt engineering decides the wording of the instruction, and sits inside the larger decision.
- Retrieve chunks rather than whole documents, because chunking a source into passages of a few hundred tokens lets a call carry the relevant paragraphs from five articles instead of the full text of one.
- Keep the last few turns verbatim, summarise the older ones, and pin the facts that must survive, such as order numbers and addresses, outside the summary.
- Reach for context engineering before fine-tuning: a context change ships in an afternoon where a fine-tune needs data, a training job and a redeploy.