SageMaker JumpStart or Bedrock for the Same Model

The situation

A platform team is picking a serving path for a new internal service. The model has been chosen. Llama 3.3 70B Instruct, based on evaluation results from the research team. Traffic projection: starts at ~5,000 requests per day, growing to ~50,000 per day over six months. Request shape: average 2,000 input tokensTokenThe unit of text an LLM actually sees – usually a short character sequence, not a whole word., 400 output tokens. Latency target: p95 under 4 seconds. Budget: flexible but accountable, the team is expected to defend the choice against cheaper options at quarterly review.

Two serving paths are on the table. SageMaker JumpStart deploys meta-textgeneration-llama-3-3-70b-instruct onto a SageMaker real-time endpoint, and the model’s deployment configurations list ml.g5.48xlarge, ml.g6.48xlarge, ml.p4d.24xlarge and ml.p5.48xlarge. Bedrock serves the same weights as meta.llama3-3-70b-instruct-v1:0, called through Converse with per-token billing and no infrastructure. Both give a 128K context window and cap output at 4K tokens, so the request shape fits either way.

Additional context: the team has three other Bedrock-hosted models in production already (Claude for conversational, Titan for embeddings, Nova Micro for classification), so Bedrock ergonomics are familiar. They do not run any SageMaker endpoints today. They do run EKS for other workloads, so ops maturity exists, but not for real-time ML serving specifically.

What actually matters

The decision is about where the line falls between “we run the model” and “AWS runs the model.”

The first decision is pricing model. Bedrock charges per token. A SageMaker endpoint charges by instance-hour, and the box bills whether or not traffic is hitting it. The break-even depends on traffic volume: low throughput favours per-token, high sustained throughput favours instance-hour.

The second is operational surface. Bedrock: none. Call the API, done. SageMaker endpoint: health checks, autoscaling policies, deployment pipelines, instance-type tuning, monitoring for memory and GPU utilisation, endpoint version management. Not crushing overhead, but real.

The third is latency and capacity control. A dedicated SageMaker endpoint has consistent latency, no shared-tenancy queueing, and scaling policies we set. Bedrock’s shared infrastructure has variable latency under global load, and for this model there is no dedicated-capacity tier at all. Provisioned ThroughputProvisioned ThroughputReserved Bedrock capacity bought by the hour for a fixed term, paid for whether traffic fills it or not. is sold against a published list of base models that covers Anthropic Claude, Cohere Embed and Llama 3.1 and 3.2, and Llama 3.3 70B is not on it; the model’s service tiers are Standard only, with no Priority, Flex or Reserved option. Capacity is whatever the account’s per-model quota in the Region allows. If latency predictability becomes a hard requirement, the answer lives on the SageMaker side.

The fourth is customisation. A SageMaker endpoint can host a fine-tuned Llama, a quantised Llama, or a custom inferenceInferenceRunning a trained model to produce output – as opposed to training it. container running vLLM with specific flags. Bedrock’s hosted Llama is as AWS configured it, no knobs. For a team running vanilla Llama 3.3 70B this doesn’t matter; for a team running AWQ-quantised weights or an LMI config with paged attention, it does.

The fifth is Region coverage, and it is narrower than reputation suggests. Bedrock serves Llama 3.3 70B in three Regions: us-east-2 directly, plus us-east-1 and us-west-2 through the US geo inference profileInference profileA Bedrock resource wrapping a model so calls to it can be tagged, routed across regions, or repointed without changing app code. us.meta.llama3-3-70b-instruct-v1:0. There is no EU or APAC option. SageMaker hosts the weights anywhere SageMaker runs. A service that needs presence outside the United States has its answer already.

The sixth is compliance and data isolation. Both paths keep traffic off the public internet: Bedrock has interface VPC endpoints through PrivateLink, and SageMaker attaches elastic network interfaces in your subnets to the model containers, so inference sits behind your security groups and route tables. The difference is where the weights run. Bedrock runs each provider’s model in a Bedrock-operated deployment account the provider has no access to, so prompts and completions never reach Meta. SageMaker runs them on instances billed to your account. Some regimes draw a line there; most draw it at the network path.

The team’s operational appetite settles the ties. A team that likes running infrastructure and values the control will pick SageMaker; a team that would rather ship features and leave the GPU fleet to AWS will pick Bedrock.

What we’ll filter on

1. Cost at projected throughput, what’s the monthly bill at 5k/day and at 50k/day?
2. Operational surface, what do we run, tune, and monitor?
3. Latency profile, p50, p95, p99 at expected load?
4. Customisation, can we run the model with the flags we want?
5. Region coverage, can we serve it where the service has to live?
6. Integration with the rest of the stack, same SDK, same IAM, same CloudWatch story?

The landscape

1. Bedrock on-demand (meta.llama3-3-70b-instruct-v1:0). USD$0.72 per million tokens, input and output alike. No infrastructure. The same Converse call as the team’s other Bedrock models, with IAM, CloudTrail and CloudWatch already wired. Response streaming, Guardrails, Agents, Flows and Prompt management all list this model; Knowledge Bases, model evaluation and structured outputs do not. Three US Regions, shared tenancy, latency good but variable.

2. Bedrock batch inferenceBatch inferenceSubmitting a bulk job of model calls to run asynchronously at a lower per-token price, trading immediacy for cost.. The same model at USD$0.36 per million tokens, half the on-demand rate, with results in hours rather than seconds. Only fits an offline share of the workload; nothing here meets a 4-second p95. It matters as a boundary marker: with no Provisioned Throughput for this model, batch is the only alternative to on-demand inside Bedrock.

3. SageMaker JumpStart deployment. Choose one of the listed instance types, deploy, get an endpoint URL. JumpStart packages LMI inference containers with working defaults, including an lmi-optimized configuration that turns on speculative decoding. Billed by instance-hour, from USD$16.688 for ml.g6.48xlarge to USD$63.296 for ml.p5.48xlarge in us-east-1. The endpoint scales via autoscaling policies we configure.

4. SageMaker with a custom container. JumpStart as the starting point, our own inference container replacing the default. This is also the route onto a smaller box than JumpStart’s own configurations offer: AWQ int4 weights fit an ml.g5.12xlarge at USD$7.09 an hour, under half the cheapest listed option. Maximum control over the inference runtime (quantisationQuantisationStoring model weights at lower precision (8 bits, 4 bits, sometimes fewer) so the model is smaller and faster to run., batching strategy, attention algorithm). Maximum ops overhead.

5. Self-hosted on EKS with vLLM/TGI. GPU nodes in EKS, a vLLM Deployment serving Llama, an internal LoadBalancer. Most flexible; highest ops cost. Suits teams with existing Kubernetes ML-serving maturity.

6. Bedrock Custom Model Import. For teams with their own modified weights: Llama 3.3 is a supported architecture, in eu-central-1, us-east-1, us-east-2 and us-west-2. Billing is per Custom Model Unit per minute (USD$0.05718 in us-east-1) over 5-minute windows from the first successful inference call, with Bedrock raising and lowering the number of running model copies as demand changes, plus USD$1.95 per model per month for storage. Batch inference is not available on imported models. Here the weights are vanilla Llama 3.3 70B, which Bedrock already serves per token, so an import adds cost and returns nothing.

Why serving an LLM is its own problem

The three self-hosted rows carry a class of work that a traditional ML endpoint never had, and it is worth naming before the numbers, because it is what the ops column is really measuring. Llama 3.3 70B at FP16 is roughly 140GB of weights, and most of the unique challenges of large language models follow from that number. Container-based deployment patterns for a gradient-boosted model tune for request concurrency; the same patterns here are optimised for memory requirements first and everything else after.

Model loading comes first. The container pulls the weights from S3 and loads them into GPU memory before it can answer anything, which is a multi-minute cold start rather than a multi-second one. Scale-from-zero, the usual reflex for a spiky endpoint, produces timeouts while a fresh instance is still reading weights off the network. Specialised model loading strategies are the answer: bake the weights into the image in Amazon ECR so startup is a layer fetch rather than a runtime download, keep a warm pool of pre-loaded instances behind the scaling policy, or hold a provisioned floor of one instance and scale above it. Either way the endpoint’s minimum size is one fully loaded model, and that minimum is the flat monthly floor in the cost comparison below. The same constraint applies to the model on Amazon ECS or EKS with GPU capacity providers; the runtime changes, the 140GB does not.

Sizing comes next, and GPU memory rather than vCPU is the binding constraint. What has to fit is weights plus the KV cache, and the cache grows with concurrency multiplied by context length, so a configuration that sits comfortably at batch size one falls over at batch size sixteen with 2,000-token inputs. GPU utilisation on a healthy LLM endpoint runs high by design, because memory left idle is memory that could have held another sequence, which makes it a poor autoscaling signal on its own.

Throughput is then measured in tokens per second rather than requests per second, and batching strategy sets it more than instance count does. vLLM’s continuous batching admits new sequences into a running batch as older ones finish, which is what keeps a 48xlarge busy at high concurrency; tensor parallelism splits a model too large for a single card across the GPUs in the instance. Token processing capacity, not invocation count, is what the scaling policy should watch. Quantisation is the last lever: AWQ int4 trades a little output quality for a smaller footprint and a cheaper box, and it is what moves 70B onto an ml.g5.12xlarge.

None of this exists on Bedrock’s on-demand path. No cold start to engineer around, no KV-cache arithmetic, no batching flags, no instance floor. AWS carries that work and prices it into the per-token rate.

Evaluation

Side by side

At 5k requests/day × 2,400 tokens average = 12M tokens/day, ~360M tokens/month. All figures are us-east-1 list prices over a 730-hour month.

Bedrock on-demand (meta.llama3-3-70b-instruct-v1:0):
  360M tokens x USD$0.72/M (input and output)     ~ USD$260/month

SageMaker ml.p4d.24xlarge:
  USD$25.2513/hour x 730 hours                    ~ USD$18,400/month

SageMaker ml.g5.48xlarge (FP16):
  USD$20.36/hour x 730 hours                      ~ USD$14,900/month

SageMaker ml.g6.48xlarge (cheapest listed):
  USD$16.688/hour x 730 hours                     ~ USD$12,200/month

SageMaker ml.g5.12xlarge (AWQ int4, custom container):
  USD$7.09/hour x 730 hours                       ~ USD$5,180/month

At 50k/day = 120M tokens/day ≈ 3.6B/month, Bedrock on-demand reaches ~USD$2,600 and the endpoint costs stay flat. That is half the hand-built quantised endpoint and a fifth of the cheapest one-click one. The break-even against the quantised ml.g5.12xlarge sits near 100,000 requests/day at this request shape; against ml.g6.48xlarge it is near 235,000, and the larger instances later still.

Cost vs throughput, plotted

The solution

Start on Bedrock, and expect to stay there for a long time.

At the starting volume (5k/day), Bedrock costs ~USD$260/month and has no operational cost. The cheapest one-click endpoint, ml.g6.48xlarge, is ~USD$12,200/month before anyone sets up monitoring or scaling. Even a hand-built quantised endpoint on ml.g5.12xlarge is ~USD$5,180/month, and that one carries the most ops work of the three. Bedrock is an order of magnitude cheaper on both axes.

At the target volume (50k/day), the arithmetic barely changes. Bedrock reaches ~USD$2,600/month, still half the quantised endpoint’s floor and a fifth of the one-click floor. On pure cost, Bedrock holds the lead well past the six-month projection, and past several years of the growth rate the team is projecting.

The migration plan. Don’t pre-optimise. Ship on Bedrock. Track usage weekly. Set the cost tripwire at 100k requests/day, where Bedrock lands near USD$5,200/month and the quantised endpoint’s USD$5,180 floor is finally level with it. That is twice the six-month target, so the honest expectation is that cost never triggers the move.

Latency considerations. Bedrock latency is fine but variable, and for this model there is no dedicated-capacity tier to fall back on: no Provisioned Throughput, no Priority or Reserved service tier, only the account’s per-model quota in the Region. A dedicated endpoint removes shared-tenancy queueing from the picture entirely, and with vLLM’s continuous batching it holds a steady p95 under concurrency that varies through the day. If the p95 SLA is strict, that predictability is a stronger migration case than the bill, and it arrives sooner. Measure Bedrock’s p95 against the 4-second target for a full month before deciding.

Region coverage. This is the one that decides the choice outright rather than tipping it. Bedrock serves Llama 3.3 70B in us-east-1, us-east-2 and us-west-2 only, through the US geo inference profile. A service that has to run in Sydney, Frankfurt or Tokyo cannot use Bedrock for this model at all, and JumpStart is the only path. Check Region coverage before the cost math, not after.

Team readiness. The team hasn’t run SageMaker endpoints before. The first real-time GPU endpoint is a learning curve: autoscaling policies, instance-type tuning, monitoring GPU utilisation against CPU memory, handling deployment rollouts. None of this is hard, but all of it is new. Running Bedrock while the team builds endpoint skills on the side is a sensible ramp.

When to pick SageMaker from day one. Three scenarios flip the default: (1) the model has to run outside us-east-1, us-east-2 or us-west-2, (2) the model needs customisation (quantisationQuantisationStoring model weights at lower precision (8 bits, 4 bits, sometimes fewer) so the model is smaller and faster to run., fine-tuneFine-tuningContinuing to train an already-trained model on a smaller dataset to adapt its behaviour., custom inference flags) that Bedrock doesn’t expose, (3) a latency SLA that shared tenancy can’t hold, with no Provisioned Throughput available to fix it. Any of those, pick SageMaker. None of them, pick Bedrock and stop thinking about it.

Worked example

Week 1: service launches on Bedrock. Llama 3.3 70B via Converse, the same SDK pattern as the other Bedrock services. ~200 requests/day during internal testing; a ~USD$10/month bill.

Weeks 2-8: traffic grows to ~4,000 requests/day as teams adopt the service. Bedrock spend ~USD$210/month. CloudWatch metrics tracking p95 latency (holding at 2.8s), invocation count, throttle rate (zero). Monthly review: on track, no migration planned.

Weeks 9-12: traffic hits ~8,000 requests/day; integration with a customer-facing product kicks in. Bedrock spend ~USD$415/month. The comparison against SageMaker shows Bedrock ahead by roughly USD$4,800/month versus the quantised endpoint and USD$11,800 versus the cheapest one-click one. No migration.

Quarter-end review: spend projection to the end of next quarter based on current growth. The team presents the cost curve, the 100k/day tripwire, and the finding that the real migration triggers are Region coverage and the p95 SLA. Finance and product align on staying with Bedrock, building endpoint skills on the side, and revisiting only if the service has to serve a non-US Region or the latency target tightens.

The decision is defensible, reversible, and produced by numbers rather than preference.

What’s worth remembering

1. Bedrock and JumpStart serve the same model through different operational surfaces. Bedrock costs per token and needs no ops; a JumpStart endpoint costs instance-hours and gives control.
2. Break-even sits far higher than intuition says. At USD$0.72 per million tokens either way, Llama 3.3 70B needs ~100k requests/day at 2,400 tokens each to match a hand-quantised ml.g5.12xlarge, and ~235k/day to match the cheapest JumpStart-listed instance.
3. Llama 3.3 70B on Bedrock is a three-Region model: us-east-2 directly, us-east-1 and us-west-2 through the US geo inference profile. Anywhere else, SageMaker is the only path, and that check comes before the cost math.
4. There is no dedicated-capacity option for this model inside Bedrock. Provisioned Throughput covers a published list that includes Claude, Cohere Embed and Llama 3.1 and 3.2, and Llama 3.3 70B is absent from it; the service tier is Standard only.
5. Customisation is what only SageMaker offers. Quantisation, a custom inference runtime and modified weights all need an endpoint, or a Custom Model Import back into Bedrock at USD$0.05718 per Custom Model Unit per minute.

Two doors, same model, different rooms behind them. The right door depends on how much infrastructure the team is willing to run, where the service has to live, and at what point in the traffic curve the math tips. Start where the bill is lowest; migrate when something other than the bill forces it.