An AI Use Case Envisioning session at Lodgewise, a residential property-management agency, produced a short portfolio of pilots. This is the first one built: triaging the inbound maintenance requests that arrive at the desk by email and web form, a few hundred a week, currently sorted by hand.
The envisioning session pinned the use case before any code was written, and those pins are the whole specification:
- Capability: classify. Put each request into a category, a priority, and a suggested trade. Not a chatbot, not an agent. The cheapest, most evaluable shape of AI there is.
- Autonomy: draft for review. The model proposes; the coordinator confirms with one click. It never dispatches a trade on its own.
- Cost of being wrong: real but bounded by the human, except for emergencies. A misfiled “leaking tap” wastes a few minutes. A missed gas leak is a different category of mistake, so emergencies are routed to a person regardless of what the model thinks.
The session also did the team a favour by sending two neighbouring ideas elsewhere: arrears prediction went to a rule, and duplicate-ticket detection went to a matching query, both off the when-not-to-use-an-llm list. What’s left is a genuine classify problem: messy human text in, a few structured fields out.
What “good” looks like
Before reaching for a model, write down the output you actually want. A maintenance request like:
“Hi, the hot water’s been off since last night and we’ve got a newborn, the tank in the laundry is making a clicking noise. Please help.”
should come back as something a desk system can route:
{
"category": "plumbing_hot_water",
"priority": "urgent",
"suggested_trade": "plumber",
"is_emergency": false,
"summary": "No hot water since last night; clicking from laundry tank. Tenant has a newborn.",
"confidence": 0.82
}
Four routing fields, a one-line summary for the human, and a confidence score the system can threshold on. The categories, priorities, and trades are a closed list the agency already uses; the model’s job is to map free text onto that list, not to invent new buckets.
That closed list matters. A classifier with an open-ended output is really a generator, and you can’t measure a generator the way you can measure a choice between known options. Pin the vocabulary first.
The classifier
Bedrock’s Converse API gives a single, model-agnostic call. For a classify job, reach for a small, fast model; this is not where you spend on the largest one (model choice is its own decision, see Picking the AWS AI Service Tier).
import json
import boto3
brt = boto3.client("bedrock-runtime", region_name="ap-southeast-2")
# A small, cheap model is the right tool for classification.
MODEL_ID = "anthropic.claude-3-haiku-20240307-v1:0"
CATEGORIES = [
"plumbing_hot_water", "plumbing_leak", "plumbing_blockage",
"electrical", "appliance", "heating_cooling", "locks_security",
"pest", "structural", "grounds_garden", "general", "unclear",
]
TRADES = ["plumber", "electrician", "appliance_tech", "handyman",
"locksmith", "pest_control", "none"]
PRIORITIES = ["emergency", "urgent", "routine", "low"]
SYSTEM = f"""You triage maintenance requests for a residential property
manager. Classify each request using ONLY these closed lists.
category: one of {CATEGORIES}
priority: one of {PRIORITIES}
suggested_trade: one of {TRADES}
is_emergency: true only for immediate risk to safety or property
(gas smell, flooding, no heating in extreme weather, exposed wiring,
break-in, anyone unsafe). When unsure whether it is an emergency,
set is_emergency true and priority emergency. Safety beats tidiness.
summary: one neutral sentence for a human coordinator.
confidence: your confidence from 0 to 1 that category and trade are right.
If the text is too vague to classify, use category "unclear",
suggested_trade "none", and a low confidence.
Reply with a single JSON object and nothing else."""
def triage(message_text: str) -> dict:
resp = brt.converse(
modelId=MODEL_ID,
system=[{"text": SYSTEM}],
messages=[{"role": "user", "content": [{"text": message_text}]}],
inferenceConfig={"temperature": 0, "maxTokens": 400},
)
raw = resp["output"]["message"]["content"][0]["text"]
return json.loads(raw)
Temperature zero, because classification wants the same answer every time, not creativity. The closed lists live in the system prompt so they’re easy to version; when the agency adds a category, you change one list and re-run the eval set rather than redeploying logic.
Structured output you can trust
json.loads on a model’s text is a promise the model can break. It might wrap the JSON in prose, invent a category outside the list, or omit a field. A classifier that occasionally returns something unparseable is worse than no classifier, because the failures are silent. Validate every response against the closed lists before anything downstream sees it:
def validate(result: dict) -> dict:
if result.get("category") not in CATEGORIES:
result["category"] = "unclear"
result["confidence"] = min(result.get("confidence", 0), 0.3)
if result.get("suggested_trade") not in TRADES:
result["suggested_trade"] = "none"
if result.get("priority") not in PRIORITIES:
result["priority"] = "urgent" # fail toward attention, not silence
result["is_emergency"] = bool(result.get("is_emergency", False))
result["confidence"] = float(result.get("confidence", 0))
return result
Note the direction of every fallback: an unrecognised category becomes unclear, an unknown priority becomes urgent, a parse failure (caught upstream) routes to a human. The classifier is allowed to be uncertain; it is never allowed to quietly drop a request on the floor.
The more robust version of this is to stop asking for JSON in free text at all and define the output as a Bedrock tool the model must call, so the API enforces the shape. That’s the same machinery as wiring a model to actions, and it’s worth adopting once the pilot proves out: wiring function calling through Bedrock. For a first pilot, prompt-and-validate is enough and keeps the moving parts visible.
Guardrails and the tenant’s data
A maintenance request is full of personal information: names, phone numbers, sometimes “I’ll be home after my chemo appointment.” None of that needs to reach the model to classify a plumbing fault, and none of it should sit in your prompt logs. Attach a Bedrock guardrail that masks PII on the way in and filters it on the way out, the same control set out in configuring Bedrock guardrails:
GUARDRAIL = {"guardrailIdentifier": "lodgewise-triage", "guardrailVersion": "3"}
def triage(message_text: str) -> dict:
resp = brt.converse(
modelId=MODEL_ID,
system=[{"text": SYSTEM}],
messages=[{"role": "user", "content": [{"text": message_text}]}],
inferenceConfig={"temperature": 0, "maxTokens": 400},
guardrailConfig=GUARDRAIL,
)
if resp.get("stopReason") == "guardrail_intervened":
return {"category": "unclear", "priority": "urgent",
"suggested_trade": "none", "is_emergency": False,
"summary": "Routed to a person (content filtered).",
"confidence": 0.0}
raw = resp["output"]["message"]["content"][0]["text"]
return validate(json.loads(raw))
When the guardrail intervenes, the request doesn’t vanish; it routes to a human with a neutral note. The PII masking also keeps the tenant’s details out of the logs you’ll be reading when you debug a misclassification, which is its own quiet benefit, and keeps you on the right side of keeping PII out of prompts and logs.
Knowing when it’s wrong
The autonomy rung from the envisioning session, draft for review, is enforced in two lines of routing logic, not in the model:
CONFIDENCE_FLOOR = 0.6
def route(result: dict) -> str:
if result["is_emergency"]:
return "human_now" # never auto-handled, any confidence
if result["confidence"] < CONFIDENCE_FLOOR:
return "human_review" # model unsure: a person decides
if result["category"] == "unclear":
return "human_review"
return "suggested" # pre-filled, coordinator confirms
Three rules carry the safety of the whole system. Emergencies always reach a person, whatever the model’s confidence. Low-confidence calls go to review rather than being acted on. Everything else is presented to the coordinator pre-filled, with the model’s category and trade selected and a confirm button, which is the difference between draft for review and act autonomously. The coordinator’s click is fast when the suggestion is right and a correction when it isn’t, and every correction is a labelled example for the next eval run.
The confidence floor is a dial, not a constant. Start it high, so the model only auto-suggests when it’s sure and humans see more, then lower it as the eval numbers earn the trust. That’s how you climb the autonomy ladder honestly: on evidence, not on the demo.
Measuring it
A classifier you haven’t measured is a rumour. Before this goes near the live desk, build an eval set: a few hundred real historical requests with the category, priority, and trade a senior coordinator agrees are correct. Lodgewise already had years of triaged tickets, which is exactly why the envisioning session scored this one feasible.
Run the classifier over the held-out set and look at the numbers that matter for this job, not at a single accuracy figure:
def evaluate(labelled):
n = len(labelled)
cat_right = sum(triage(x["text"])["category"] == x["category"]
for x in labelled)
# The metric that actually matters: did we ever call a real
# emergency non-urgent? That is the failure with a body.
missed_emergencies = sum(
1 for x in labelled
if x["is_emergency"] and not triage(x["text"])["is_emergency"]
)
print(f"category accuracy: {cat_right / n:.1%}")
print(f"missed emergencies: {missed_emergencies} of {n}")
Overall category accuracy is the headline, but the number that decides whether this ships is missed_emergencies. A model that’s 94% accurate on category but once labelled a gas smell “routine” doesn’t go live; the prompt that says safety beats tidiness, when unsure mark it an emergency exists precisely to push that count to zero, accepting some false alarms as the price. False emergencies cost a coordinator a glance; missed ones cost a great deal more, so the trade is deliberately lopsided. For the broader machinery of scoring model output as a repeatable job, see evaluating LLM output with Bedrock eval jobs.
Wiring it into the desk
The finished pilot is unglamorous, which is the point. A request arrives; a small Lambda runs triage, validate, and route; the result is written next to the ticket in the system the coordinators already use. A confident, non-emergency call shows up as a pre-filled suggestion with a confirm button. A low-confidence or emergency call shows up flagged for a person, with the model’s summary as a starting point rather than a decision.
No part of this dispatches a trade, emails a tenant, or closes a ticket. The model reads and suggests; the human still owns every consequence. That restraint is what makes the pilot safe to run on real tenants in week one instead of after a quarter of nervous meetings.
What it bought, and the next rung
After a month, the desk is sorting requests in a fraction of the time, the coordinators spend their attention on the genuinely ambiguous ones, and every confirmation and correction is quietly building a better-labelled dataset than the agency has ever had. The eval numbers are the asset: they’re what licenses the next move.
The next rung is not “let it dispatch trades.” It’s narrower and earned: for the two or three highest-volume, lowest-risk categories where the model has proven near-perfect on the eval set, raise the confidence floor’s upper band so those go straight to a work order, while everything else stays draft-for-review. You climb the ladder one well-measured category at a time. That, and not a bigger model, is what turns a triage pilot into a triage system, and it sets up the agency’s second pick from the same session: answering tenant questions from the lease, a different capability family with a different rung.