The situation
An engineering firm runs most workloads on AWS and keeps three things elsewhere. A pair of on-premises domain controllers still authenticate staff to legacy systems. Identity for everything else is a SaaS identity provider. And a payment vendor’s console, used by four people in finance, holds an audit log of who approved what.
An incident review found the gap. An account compromise was investigated using CloudTrail, which showed what the compromised role did in AWS, and could not show how the credential was obtained, because that happened at the identity provider. The investigator asked the identity provider for logs, waited two days for a CSV, and correlated it by hand in a spreadsheet.
The requirement is that the next investigation does not need a spreadsheet. Records from the three external sources should be queryable alongside AWS logs, retained under the same policy, with the same integrity guarantees.
What actually matters
The first thing that matters is that ingesting logs is easy and joining them is not. Three sources arriving in three formats, with three different names for a user and three different timestamp conventions, produce a data lake nobody queries during an incident because correlating it takes longer than asking the vendor. The normalisation decision matters more than the transport.
The second is that the sources differ in what they will give you. A machine you control can run an agent. A SaaS product gives you whatever API or export it chooses to offer, on its schedule, and may charge for it. That difference decides the mechanism per source and there is no single answer for all three.
The third is that latency requirements differ by use. Detection wants events in near real time so a rule can fire; investigation wants completeness and tolerates delay. A design optimised for one is often wasteful for the other, and being clear which sources need which avoids building a streaming pipeline for a log four people generate a week.
The fourth is that once a record is in your estate it should inherit the estate’s controls: the same retention, the same object lock, the same access policy. A third-party log in a bucket with different rules is a gap in the compliance story rather than an addition to it.
Underneath it, the identity provider is the one that matters most, because it is where the credential was obtained and it is where the next investigation will start.
What we’ll filter on
- Can we run an agent on it, or are we limited to what its API offers?
- What format does it emit, and what does normalising it cost?
- Does this source need near-real-time delivery, or is a daily batch sufficient?
- Can records be joined to AWS records on an identity or a request?
- Does it inherit the estate’s retention and integrity controls?
- What happens when the vendor changes the format?
The landscape
Systems Manager hybrid activation. Registers an on-premises machine as a managed node, so it can run the CloudWatch agent, receive State Manager associations and Run Command, and appear in the same inventory as EC2. For the domain controllers this turns “a server we cannot reach” into a managed node that ships logs the same way everything else does.
The CloudWatch agent. Collects Windows event logs, syslog and arbitrary files, and ships them to CloudWatch Logs. On a domain controller this is the mechanism for security event logs, and the collection configuration is what decides whether you get the events that matter or all of them.
Kinesis Data Firehose. The general-purpose ingestion path for anything that can make an HTTP request: buffered delivery to S3, with optional transformation through Lambda on the way. This is the transport for a SaaS product that can post to a webhook, and the Lambda is where normalisation happens.
Security Lake custom sources. Security Lake accepts custom sources alongside the AWS-native ones, provided the data is written as OCSF-formatted Parquet into the lake’s structure. This is the mechanism that solves the joining problem: an identity provider’s sign-in event and a CloudTrail event both become OCSF records with a common shape, and a query spans them without a correlation script.
OCSF. The Open Cybersecurity Schema Framework, the normalised schema Security Lake uses. Mapping a source to it is real work, done once per source, and it is what converts three formats into one queryable set. This is the decision that determines whether the lake is used during an incident.
Third-party integrations into Security Hub. Many security vendors publish findings directly into Security Hub in the AWS Security Finding Format. Where a source is a security product rather than a raw log, this is a shorter path than building a pipeline, and it produces findings rather than events.
AppFabric. Ingests audit logs from supported SaaS applications and normalises them to OCSF, delivering to S3 or Firehose without a custom integration per vendor. Where a SaaS product is on its supported list, it removes most of the work described above.
EventBridge partner event sources. Some SaaS vendors publish events into EventBridge directly, which gives near-real-time delivery and rule-based routing without polling an API.
A polling Lambda. The fallback for a vendor with an API and nothing else: a scheduled function pulling since a cursor, writing to Firehose. It is the most work and the most maintenance, and it is what a vendor without an integration leaves you.
Evaluation
Side by side
| Source | Mechanism | Latency | Normalisation | Maintenance |
|---|---|---|---|---|
| On-premises domain controllers | SSM hybrid activation + CloudWatch agent | Near real time | Custom mapping to OCSF | Agent lifecycle |
| SaaS identity provider (supported) | AppFabric | Minutes | Done for you | Low |
| SaaS identity provider (partner event source) | EventBridge | Near real time | Custom mapping | Low |
| SaaS identity provider (API only) | Scheduled Lambda + Firehose | Batch | Custom mapping | Yours forever |
| Payment vendor console | Scheduled Lambda or manual export | Daily | Custom mapping | Yours forever |
| Security product findings | Security Hub integration | Minutes | ASFF, done for you | Low |
The column that decides the effort is normalisation, and the rows where somebody else has done it are worth a great deal. Checking AppFabric’s supported list and the vendor’s Security Hub or EventBridge integrations before writing anything is the step that most often removes the pipeline entirely.
The solution
Check for an existing integration first, use hybrid activation for the machines, normalise everything to OCSF into Security Lake, and match the delivery mechanism to whether the source drives detection or investigation.
Start by checking what already exists, because the cheapest pipeline is one somebody else maintains. Look for the identity provider on AppFabric’s supported applications, for a partner event source in EventBridge, and for a Security Hub integration. Teams routinely build a polling Lambda for a product that had an integration, and the cost of checking is an afternoon.
The domain controllers get Systems Manager hybrid activation, which registers them as managed nodes and brings them into the same operational surface as everything else. The CloudWatch agent then ships the Windows security event log, with a collection configuration naming the event IDs that matter rather than everything, because a domain controller’s security log at full verbosity is an enormous and mostly uninteresting stream. State Manager holds the agent configuration in place, so a rebuilt controller comes back instrumented.
The identity provider is the priority, because it is where the last investigation stopped and where the next will start. If AppFabric supports it, that is the answer and normalisation comes free. If not, EventBridge partner events for near-real-time delivery, or a polling Lambda into Firehose as the fallback, with a Lambda transformation mapping to OCSF on the way through.
The payment vendor is the low-value, low-volume source and should be treated as such. Four people generating a handful of approvals a week does not justify a streaming pipeline. A scheduled daily pull into the same Firehose, or even a scheduled export, is proportionate, and being willing to say so keeps the effort where it matters.
Land everything in Security Lake as custom sources, in OCSF. This is the work, and it is what makes the lake worth having: a query joining a sign-in at the identity provider to an AssumeRole in CloudTrail to an object read in S3 becomes one statement rather than three exports and a spreadsheet. Do the mapping per source, review it against a real incident’s questions, and version it.
Everything then inherits the estate’s controls automatically, because Security Lake writes into S3 buckets you own, under the lifecycle, object lock and access policies already in place. That is the compliance half of the requirement satisfied by construction rather than by a separate design.
Then wire detection to the sources that support it. Rules on identity provider events, impossible-travel or a burst of failed authentications, delivered as findings into Security Hub alongside GuardDuty’s. The payment vendor’s daily batch drives no detection and nobody should pretend otherwise.
Finally, plan for the format changing. A vendor that alters its export breaks the mapping silently, producing records that parse into the wrong fields. A schema check on ingestion, and an alarm on a sudden drop in successfully-mapped records, is what turns that into a notification rather than a discovery during the next incident.
Why not ship everything raw into S3 and normalise at query time. It is faster to build and it moves the work into the moment when speed matters most, which is during an incident, by whoever is on call.
Why not build one polling Lambda pattern and use it for all three. It works, it is more code than necessary for two of the three, and it means owning three integrations that vendors change without telling you.
Worked example
The check for existing integrations pays for itself in an afternoon: the identity provider is on AppFabric’s supported list, which removes the pipeline and the mapping for the most important source. Nobody had looked.
The domain controllers take longer than expected, entirely on the hybrid activation prerequisites: an IAM role, an activation code with an expiry that lapses while the change is in review, and a proxy configuration for outbound access. Once registered, the agent rollout is a State Manager association like any other.
The event ID selection matters more than anticipated. The first configuration ships the whole security log and produces 40 GB a day from two machines. Narrowing to the authentication, privilege-use and account-management event IDs takes it to under 2 GB with nothing an investigation would want removed.
The payment vendor has an API with a cursor and no integration, so it gets the scheduled Lambda. It is 60 lines and runs daily, and the decision not to make it real-time is recorded so the next reviewer does not treat it as an oversight.
The OCSF mapping for the domain controller events takes a week, mostly spent deciding how a Windows account name joins to an IAM principal. The answer is a mapping table maintained alongside the pipeline, which is unsatisfying and is the actual work of correlating identity across two systems.
Four months later there is another incident. The query that took two days and a spreadsheet takes nine minutes: a sign-in from an unusual location at the identity provider, a session, an AssumeRole, and the object reads that followed, in one result set ordered by time.
The format-change alarm fires once, in the seventh month, when the payment vendor adds a field and renames another. Records were still landing; a third of the fields were empty. Catching it in a week rather than at the next investigation is the whole reason the check exists.
What’s worth remembering
- Ingesting external logs is easy and joining them is not; the normalisation decision determines whether the lake gets queried during an incident or bypassed for a vendor export.
- Check for an existing integration before building a pipeline: AppFabric for supported SaaS, EventBridge partner event sources, and Security Hub integrations for security products all remove work somebody else maintains.
- Systems Manager hybrid activation brings an on-premises machine into the same managed-node surface as EC2, so it can run the CloudWatch agent and receive State Manager associations like anything else.
- Match the mechanism to the use: detection needs near-real-time delivery, investigation tolerates a daily batch, and building a streaming pipeline for a source generating a few events a week is effort in the wrong place.
- Landing everything in Security Lake as OCSF custom sources makes external records inherit the estate’s retention, object lock and access policies by construction.
- A vendor changing its export format breaks a mapping silently, so alarm on a drop in successfully-mapped records rather than discovering empty fields during an investigation.