Of the ideas the envisioning session at Lodgewise parked, reading inspection photos was the one the room most wanted to build and most clearly couldn’t yet. A routine inspection is forty photos and an hour of a property manager writing up what they show. A model that read the set and drafted the findings would save real time. But the parking note was blunt: thousands of photos, almost none labelled with what was actually wrong, so no way to know whether the model was any good. And this is a domain where being wrong has a cost with a name on it, because “damage” in an inspection can become a deduction from a tenant’s bond.
So the build holds two things at once: bootstrap the missing labels, and never let an unreviewed model judgement reach a tenant’s money.
What good looks like
Per photo, a short structured finding, the same closed-output discipline as the triage classifier, with the one field that governs everything else, severity, kept honest:
{
"issue_type": "water_damage",
"location_hint": "ceiling above window, far wall",
"severity": "investigate",
"safety_flag": false,
"confidence": 0.71,
"note": "Brown staining and a bubble in the paint, consistent with a leak."
}
issue_type comes from a closed list (wear, water damage, mould, breakage, missing item, pest sign, safety hazard, none). severity is deliberately not a money word: cosmetic, investigate, safety, never “chargeable.” Whether anything is deducted is a human decision the model is not allowed to pre-empt, and keeping that out of the vocabulary keeps it out of the workflow.
Calling a multimodal model
The call is the same multimodal Converse shape as the multi-modal assistant, one photo at a time so each finding cites a specific image:
import json, boto3
brt = boto3.client("bedrock-runtime", region_name="ap-southeast-2")
MODEL_ID = "anthropic.claude-3-5-sonnet-20240620-v1:0"
SYSTEM = """You help a property manager review inspection photos. For the
image, report any maintenance issue you can see. Use issue_type from:
wear, water_damage, mould, breakage, missing_item, pest_sign,
safety_hazard, none. severity from: cosmetic, investigate, safety.
Set safety_flag true for anything that could be a hazard to a person
(exposed wiring, gas appliance, structural, mould at scale). When unsure,
prefer investigate over cosmetic and lower your confidence. You describe
what you see; you never decide costs or blame. Reply with one JSON object."""
def read_photo(image: bytes) -> dict:
resp = brt.converse(
modelId=MODEL_ID,
system=[{"text": SYSTEM}],
messages=[{"role": "user", "content": [
{"image": {"format": "jpeg", "source": {"bytes": image}}},
{"text": "What maintenance issues, if any, are visible?"},
]}],
inferenceConfig={"temperature": 0, "maxTokens": 400},
)
return json.loads(resp["output"]["message"]["content"][0]["text"])
The instruction to prefer investigate over cosmetic when unsure, and to flag safety generously, deliberately tilts the model toward false alarms over misses. A false alarm costs a property manager a second glance; a missed safety hazard or a missed leak costs a great deal more, so the trade is lopsided on purpose.
Bootstrapping the labelled set
Here’s the move that unblocks the parking note. The thing standing in the way was the absence of labels, and the model is a label-suggesting machine. So run it across the inspection backlog in suggest mode, present each finding to the property manager as a pre-filled checkbox they confirm, correct, or reject, and capture every one of those human verdicts:
def review_queue(photo_set):
for image in photo_set:
finding = read_photo(image)
# The human sees the suggestion and returns a verdict.
# verdict in {"confirm", "correct", "reject"}; corrections carry
# the right label. Every verdict is stored as ground truth.
yield {"image": image, "suggested": finding}
Within a few weeks of normal inspections, the agency has what it never had: a growing set of photos with confirmed labels, built as a by-product of work the property managers were doing anyway. That set is the eval set the parking note demanded, and it arrives without a separate labelling project. The flywheel from keeping an AI pilot honest is here doing double duty: it bootstraps the measurement and keeps it fresh.
The safety floor and the bond
Two rules sit above the model and never yield to its confidence. Anything with safety_flag true goes to a person immediately, regardless of how sure or unsure the model was, because a missed hazard is the one failure with a body. And nothing the model produces is ever attached to a bond deduction without a property manager confirming it against the move-in condition report, because a confident “damage” on what was pre-existing wear is exactly the mistake that turns a time-saver into a dispute. The model drafts the inspection findings; the human owns every consequence that reaches the tenant.
Measuring it
Score the model against the bootstrapped labels, and watch the asymmetric metrics, not a single accuracy figure:
def evaluate(labelled):
n = len(labelled)
missed_safety = sum(1 for x in labelled
if x["true_safety"] and not read_photo(x["image"])["safety_flag"])
missed_issue = sum(1 for x in labelled
if x["true_issue"] != "none"
and read_photo(x["image"])["issue_type"] == "none")
print(f"MISSED SAFETY: {missed_safety} of {n}") # must trend to zero
print(f"missed issues: {missed_issue} of {n}")
print(f"false alarms tolerated as the price of the above")
MISSED SAFETY is the number that can hold the pilot back on its own, the same way missed emergencies gate the triage classifier and isolation leaks gate tenant Q&A. A photo reader that drafts beautiful reports but once waved past exposed wiring does not ship. Precision on the cosmetic stuff can be mediocre and the pilot is still a win, because a human is reading the draft anyway; recall on safety cannot be.
What it bought
The parked idea became a pilot not by waiting for a labelling budget but by using the model to earn its own ground truth under human supervision. Property managers now get a drafted set of findings to edit rather than a blank page, the genuinely ambiguous and the genuinely dangerous photos are surfaced for their attention, and every inspection quietly improves the eval set. All three of the envisioning session’s AI picks are now live or unblocked, each pinned to the autonomy rung its cost-of-being-wrong earned, and each with a human in exactly the place a wrong answer would otherwise have hurt.