Turning On Data Events Without a Surprise Bill

The situation

An analytics company had an incident. A departing employee downloaded a substantial number of customer files from S3 in their final week, and the investigation could not establish which objects, because CloudTrail data events were not enabled. Management events showed the bucket policy had not changed and nothing else.

The immediate reaction was to enable data events for all S3 buckets and all Lambda functions across the organisation. That ran for nine days before finance escalated: the estate includes a data lake where a nightly job performs tens of millions of GetObject calls, and the CloudTrail charge for those nine days was AUD$47,000.

Data events are now off again, which restores the original problem. The requirement is to be able to answer “who read this object” for the data that matters, at a cost the business will accept, and to be able to demonstrate the coverage to an auditor.

What actually matters

The first thing that matters is the difference between the two event categories, because it decides everything downstream. Management events record control-plane operations and the first copy is delivered at no charge to a trail. Data events record data-plane operations, are not free, and are the only ones that answer what was read.

The second is that data events are selectable, and selection is the entire skill here. Advanced event selectors filter on the resource, the event name, the resource ARN prefix, and whether the call was read or write, which means “log reads of the customer bucket and nothing else” is expressible. The nine-day bill came from not selecting.

The third is that volume follows the workload rather than the data’s sensitivity. The data lake generating tens of millions of reads holds derived data nobody would investigate; the customer bucket generating thousands holds the data the incident was about. Cost tracks the first and value tracks the second, which is why an unselected trail is the worst of both.

The fourth is that read and write events have different value. A write is comparatively rare and always interesting. A read is common and interesting only on sensitive data. Splitting on readOnly is often the single most effective selector available.

Underneath it, CloudTrail is not the only mechanism, and for some questions it is not the best one. S3 server access logs answer a similar question at a different price and latency, and the choice between them is worth making deliberately.

What we’ll filter on

1. Does the question need control-plane or data-plane evidence?
2. Which specific resources hold data worth investigating?
3. Are reads interesting here, or only writes?
4. What is the call volume on this resource, and therefore the cost?
5. How quickly must the record be available?
6. Does the coverage need to be demonstrable to an auditor?

The landscape

Management events. Control-plane operations: creating a bucket, changing a policy, assuming a role. The first copy delivered to a trail in each account is free, which is why an organisation trail for management events is the uncontroversial baseline. They cannot tell you an object was read.

Data events. Data-plane operations: GetObject, PutObject, DeleteObject, Lambda Invoke, DynamoDB item operations, and a growing list of other resource types. Charged per event delivered. These are the only CloudTrail record of what was read.

Advanced event selectors. The selection mechanism, and the answer to the cost problem. They filter on eventCategory, resources.type, resources.ARN (with prefix matching), eventName, and readOnly. A selector saying “S3 data events, where the ARN starts with the customer bucket’s prefix, and readOnly is either value” logs what matters and excludes the data lake entirely.

Basic event selectors. The older mechanism, coarser: bucket-level selection without the ARN prefix matching and without eventName filtering. Named because a scenario may present them, and advanced selectors are what you want.

S3 server access logs. S3’s own logging of requests, delivered to a bucket as files, with no per-event charge beyond the storage and the requests to write them. Delivery is best-effort and can take hours, and the format is different. For high-volume buckets where CloudTrail data events are unaffordable, this is the cheaper record, at the cost of latency and a weaker delivery guarantee.

CloudTrail Lake. An event data store with SQL querying and long retention, which can ingest data events with the same selectors. It changes where the events go and how they are queried rather than whether they cost.

Security Lake. Normalises CloudTrail and other sources into OCSF in your own S3, which suits analysis across sources and does not change the selection decision.

S3 Storage Lens and metrics. Aggregate request counts by bucket and prefix, without per-request detail. Useful for the sizing exercise before enabling anything: knowing which buckets generate millions of requests is what stops the nine-day surprise.

GuardDuty S3 Protection and Macie. Different questions. GuardDuty analyses data events for anomalous access patterns and generates findings; Macie discovers and classifies sensitive data. Neither replaces the audit record, and GuardDuty’s S3 Protection consuming data events is worth knowing about when reasoning about what is already being generated.

Evaluation

Side by side

Source	Answers “who read it”	Cost driver	Latency	Selectable
Management events	✗	Free for the first copy	Minutes	By nothing needed
Data events, unselected	✓	Per event, all of them	Minutes	✗ by definition
Data events, advanced selectors	✓ for selected resources	Per event, selected only	Minutes	✓ ARN prefix, eventName, readOnly
S3 server access logs	✓	Storage and write requests	Hours, best-effort	Per bucket
CloudTrail Lake	✓	Ingestion and retention	Minutes	Same selectors

The middle two rows are the same feature used two ways and the difference between them is a AUD$47,000 nine days. The bottom row of interest is the server access logs, which is the option to reach for when a bucket’s volume makes data events unaffordable and the latency is acceptable.

The solution

Size it first, then select on the resources that hold data worth investigating, split on read versus write, and use server access logs where volume makes data events unaffordable.

Size before enabling anything, because the previous attempt failed on a volume nobody had measured. S3 Storage Lens and the request metrics give per-bucket request counts, which turns “enable data events” from a switch into an arithmetic problem with an answer. Do the same for Lambda invocation counts, since a high-throughput function generates data events at the same rate it runs.

Then classify the buckets. Which hold data that would be the subject of an investigation, and which hold derived, public, or operational data that would not. This is a security conversation and it is the one that determines the bill, which is worth saying to whoever objects to spending time on it.

Configure the trail with advanced event selectors scoped to the classified set: S3 data events where the resource ARN starts with each sensitive bucket’s prefix. Everything else, including the data lake with its tens of millions of reads, is excluded by not being named. This alone would have turned the AUD$47,000 into a fraction of it.

Split on readOnly where the volume justifies it. On buckets where writes are the interesting event and reads are routine and voluminous, select writes only. On the customer bucket, where the incident was a read, select both. Making that decision per bucket rather than globally is what keeps the cost proportionate to the value.

For the high-volume buckets that are excluded but not entirely uninteresting, turn on S3 server access logs. They give a record of requests without the per-event charge, at the cost of hours of latency and best-effort delivery, which is an acceptable trade for a bucket nobody expects to investigate urgently.

Then make the coverage demonstrable. A Config rule checking that data events are enabled for buckets carrying a sensitivity tag turns the design into something continuously verified, and it means a new bucket tagged sensitive without data events is a finding rather than a gap discovered during the next incident. This is also the artefact the auditor wants.

Set a budget alert on the CloudTrail cost line specifically, not on the account total. The previous failure ran for nine days before anybody noticed, and a per-service alert would have caught it on day one.

Why not enable everything and sample. CloudTrail does not sample data events, so the choice is which resources rather than what fraction, and selection is the mechanism the service actually provides.

Why not rely on server access logs everywhere and skip data events. They are cheaper and they are best-effort with hours of latency, which is a poor foundation for the one bucket an investigation will centre on. Use them for the tail, not the core.

Worked example

Sizing takes a day. Storage Lens shows the data lake at roughly 41 million requests a day and the customer bucket at 18,000. The ratio explains the AUD$47,000 immediately, and it is the number that ends the argument about whether selection is worth the effort.

Classification finds nine buckets out of 140 that hold data an investigation would centre on. Four more are ambiguous and are resolved by asking whether their contents would appear in a breach notification.

The selectors go on with ARN prefixes for those nine. The first month’s CloudTrail data event charge is AUD$310.

The readOnly split applies to three of the nine, where reads are a routine part of a pipeline. Those keep write-only selection, and the reasoning is recorded in the same document as the classification, because the next reviewer will ask.

Server access logs go on for the data lake and six other high-volume buckets. The storage cost is modest and the records land in the archive bucket with the seven-year lifecycle already in place.

The Config rule catches its first gap five weeks later: a new bucket tagged sensitivity: high by a team following the standard, with no selector because the trail’s selectors name ARNs explicitly. That prompts a change from naming buckets to naming a shared prefix convention, so a correctly-named bucket is covered on creation.

Six months on, a genuine investigation runs. It takes eleven minutes to establish which objects a departing employee read, using CloudTrail Lake over the selected events. That is the capability the whole exercise was for, and it costs AUD$310 a month rather than AUD$150,000 a year.

What’s worth remembering

1. Management events are control-plane and free for the first copy to a trail; data events are data-plane, charged per event, and the only CloudTrail record of what was read.
2. Data events are selectable and selection is the whole skill: advanced event selectors filter on resource type, ARN prefix, eventName and readOnly, so “log reads of this bucket only” is expressible.
3. Cost tracks call volume and value tracks data sensitivity, and those rarely coincide, which is why an unselected trail pays the most for the least.
4. Splitting on readOnly is often the largest available reduction, because writes are rare and always interesting while reads are voluminous and interesting only on sensitive data.
5. Size the request volume before enabling anything; S3 Storage Lens turns “enable data events” from a switch into an arithmetic problem with an answer.
6. S3 server access logs record requests without a per-event charge, at hours of latency and best-effort delivery, which suits the high-volume tail and not the bucket an investigation will centre on.