The situation
A gaming company’s security team discovered during an unrelated audit that VPC flow logs for one production VPC had not been delivering for eleven weeks. The delivery role’s permissions had been narrowed during an unrelated IAM cleanup, the flow log entered a failed state, and nothing said so.
Reviewing further, the team found three more gaps. A CloudTrail trail in one member account had been stopped, by an account administrator who no longer works there, four months ago. An EC2 instance’s log agent had been failing to start since a base image change, so 40 instances were shipping nothing. And GuardDuty was suspended in two accounts that had been created outside the account factory.
None of these produced an alert. The dashboards were quiet, because a dashboard fed by nothing draws a flat line at zero and a flat line at zero looks like a quiet week.
The requirement now is to detect the absence of expected telemetry, across an organisation of 60 accounts, without building something that pages every time a low-traffic VPC has a quiet hour.
What actually matters
The first thing that matters is that absence and error are different signals. A misconfigured flow log reports a failed delivery status somewhere; an agent that never starts reports nothing at all. Controls that watch for errors will catch the first class and miss the second, and the second is the one an attacker creates deliberately.
The second is that “expected” has to be defined before absence can be detected. Detecting that a VPC has no flow log requires knowing that it should have one, which is a statement about policy rather than about telemetry. This is why configuration-level checks and data-level checks are both needed: one knows the log should exist, the other knows whether data is arriving.
The third is that the threshold problem is real. A rule that pages when a log group receives no events in an hour will page constantly for a low-traffic environment, and a rule loose enough to be quiet will take days to notice a real gap. Different sources need different windows, and some need a rate comparison rather than an absolute.
The fourth is that the controls should be hard to turn off, because the thing being defended against includes somebody turning things off. A detection running in the account it watches, under permissions that account controls, is a detection an account administrator can disable in the same action that creates the gap.
Underneath it, prevention beats detection here. An SCP that stops a member account disabling its trail removes a whole class of gap, and no amount of monitoring is as good as the action being impossible.
What we’ll filter on
- Does this catch a configuration change, missing data, or both?
- Is “expected” defined somewhere the check can read?
- Where does the check run, and can the watched account disable it?
- Does it produce false positives on quiet periods?
- How long between the gap starting and the alert?
- Could this have been prevented rather than detected?
The landscape
Service control policies. Prevention. An SCP denying cloudtrail:StopLogging, cloudtrail:DeleteTrail, guardduty:DeleteDetector and their relatives, applied to every account below the organisation root, means a member account administrator cannot create the gap. This removes the CloudTrail and GuardDuty cases from the detection problem entirely.
Organisation trails and delegated administration. A trail created at the organisation level applies to every account including new ones, and member accounts cannot modify or delete it. Delegated administration does the same for GuardDuty, Security Hub, Config and others: enabled centrally, with auto-enable for new accounts, so an account created outside the factory is still covered.
AWS Config rules. Configuration-level detection: cloudtrail-enabled, vpc-flow-logs-enabled, guardduty-enabled-centralized and the rest, evaluated continuously across the organisation through a conformance pack. These know what should exist because the rule says so, and they detect the flow log that was never created as well as the one that was deleted. They do not know whether data is actually arriving.
CloudWatch metric filters and alarms on absence. For a log group, an alarm on IncomingLogEvents treating missing data as breaching is the data-level check. The treatMissingData setting is the specific thing to get right: the default of missing means an alarm on a metric that stops being published never fires, which is precisely the failure being detected.
CloudWatch canaries for telemetry. A synthetic event generated on a schedule and then asserted to have arrived. This converts absence detection into presence detection: something known is emitted every five minutes, and its non-arrival is unambiguous rather than a judgement about traffic levels.
EventBridge on service events. Rules matching CloudTrail’s own StopLogging and DeleteTrail events, GuardDuty suspension, and Config recorder stops, delivering to a security account. Fast, and it depends on the very trail that might have been stopped, which is an argument for the organisation trail rather than against the rule.
Security Hub. Aggregates findings from Config, GuardDuty and the security standards, and its own controls include several about logging being enabled. It is the surface where “which accounts are not logging” becomes one list rather than 60 checks.
CloudTrail log file validation and digest files. Detects modification or deletion of delivered files, and the absence of a digest file for a period is itself the signal that delivery stopped. It is the integrity check that doubles as a gap check for that one source.
Evaluation
Side by side
| Control | Catches config change | Catches missing data | Runs where | Watched account can disable |
|---|---|---|---|---|
| SCP denying stop/delete | Prevents it | ✗ | Organisation | ✗ |
| Organisation trail | Prevents it | ✗ | Management | ✗ |
| Config rules in a conformance pack | ✓ | ✗ | Per account, centrally deployed | ✗ if delegated |
| Alarm on IncomingLogEvents | ✗ | ✓ | Per account | ✓ unless centralised |
| Synthetic telemetry canary | ✗ | ✓ unambiguously | Anywhere | Depends |
| EventBridge on service events | ✓ fast | ✗ | Per account, forwarded | Partly |
| Digest file absence | ✗ | ✓ for CloudTrail | Central bucket | ✗ |
The table splits into prevention, configuration detection and data detection, and the four gaps map onto three different rows. The stopped trail and the suspended GuardDuty are prevention problems. The missing flow log is a configuration detection problem. The dead agent on 40 instances is a data detection problem, and it is the only one of the four that a configuration check would never have found, because the flow log configuration was fine and the data was not.
The solution
Prevent what can be prevented, detect configuration with Config, detect data with absence alarms, and put a canary behind the sources that matter most.
Start with prevention, because it removes two of the four gaps permanently. An SCP denying the stop and delete actions for CloudTrail, Config, GuardDuty and Security Hub, applied to every organisational unit below the root, means a member account administrator cannot create those gaps at all. Pair it with an organisation trail and delegated administration with auto-enable for new accounts, so an account created outside the factory arrives covered rather than being discovered later.
Then deploy a conformance pack across the organisation with the logging rules: trail enabled, flow logs enabled per VPC, Config recorder running, GuardDuty enabled, log file validation on. This catches the flow log that was never created and anything that drifts out of policy, and because it is deployed from a delegated administrator account the member cannot disable it. Attach remediation where the correction is safe to automate, so a missing flow log is created rather than reported.
Data-level detection is the part most teams skip and the one that would have caught the dead agent. For each log group that matters, an alarm on IncomingLogEvents with treatMissingData set to breaching, because the default silently does nothing in exactly this case. Set the evaluation window per source rather than globally: a busy production log group can alarm on an hour of silence, and a low-traffic environment needs six or twelve.
Where the source matters enough that a threshold argument is not good enough, add a canary. A scheduled Lambda writing a known event into the pipeline every five minutes, and an alarm asserting it arrived, turns a judgement about traffic levels into a binary. Do this for the audit-relevant sources rather than for everything, since it is a component per source.
Route all of it to one place. Security Hub in a delegated administrator account, receiving Config findings, GuardDuty findings and custom findings from the absence alarms, so “which accounts are not logging” is a query rather than an investigation. Add EventBridge rules on the service events themselves for speed, forwarded to the security account, so a stop that somehow gets through the SCP is known in seconds rather than at the next Config evaluation.
Then check the checkers. A quarterly exercise that deliberately breaks one source in a non-production account and measures how long detection takes is what stops this design decaying the way the last one did. Eleven weeks was the detection time before; the exercise is how you find out what it is now.
Why not rely on Config rules alone. They are the obvious answer and they evaluate configuration. The flow log with correct configuration and a broken delivery role, and the 40 instances with a dead agent, both pass every configuration check while producing nothing.
Why not alarm on everything with a tight window. It pages on quiet weekends in low-traffic environments, and an absence alarm that cries wolf gets muted, at which point the control exists on paper only.
Worked example
The SCP goes on first and immediately blocks one legitimate action: a team decommissioning an old account cannot delete its trail. The break-glass path is an exception at the organisational-unit level for accounts in a decommissioning OU, which is a better answer than an exception per request.
The conformance pack finds the missing flow log within an hour of deployment, plus four more nobody knew about, and two accounts with the Config recorder itself stopped, which is the check checking its own prerequisite.
The absence alarms are the fiddly part. The first version uses a one-hour window everywhere and pages nine times over a weekend from development log groups. Tuning to per-source windows, and moving development environments to a six-hour window with a lower-priority route, settles it at roughly one alert a fortnight, all of them real.
The canary is deployed for CloudTrail delivery and the security agent pipeline only. Its first genuine catch is three months later: a Firehose delivery stream hitting a quota after a traffic increase, dropping records silently. Nothing else in the design would have seen that, because the configuration was correct and the volume drop was within the tolerance of the absence alarm.
The quarterly exercise, run for the first time in April, breaks the agent on a non-production instance fleet. Detection takes 74 minutes against an eleven-week baseline. The second exercise breaks a delivery role’s permissions, which takes 4 hours because the alarm window on that source was generous, and the window is tightened.
What’s worth remembering
- Absence produces no error and no alarm, and a dashboard fed by nothing draws a flat line that looks like a quiet week; detecting it needs controls that fire on the lack of something.
- Configuration checks and data checks catch different failures: a flow log with correct configuration and a broken delivery role passes every Config rule while producing nothing.
treatMissingDataon a CloudWatch alarm must be set tobreachingfor absence detection, because the default means an alarm on a metric that stops being published never fires.- Prevent what you can: an SCP denying the stop and delete actions, plus an organisation trail and delegated administration with auto-enable, removes whole classes of gap rather than detecting them.
- Run the detection somewhere the watched account cannot disable it, because the threat model includes an administrator turning things off.
- A synthetic canary converts absence detection into presence detection, which removes the threshold argument for the few sources where a threshold argument is not good enough.