How to Enforce Column and Row Access on a Data Lake

The situation

A data platform team runs a 12 TB S3-backed data lake. Forty tables across four domains (sales, marketing, inventory, finance) all registered in the AWS Glue Data Catalog. Analysts query through Athena, Redshift Spectrum, and an EMR Spark cluster. Access today is IAM-on-S3: each analyst group has an IAM role with s3:GetObject scoped to one or more bucket prefixes. A sales analyst’s role allows arn:aws:s3:::acme-lake/sales/*. Nothing finer than that.

Three requirements land at the same all-hands. sales.transactions contains a commission_amount column that drives the sales team’s pay; finance auditors need to see it, ordinary sales analysts must not, and right now both groups read the same Parquet files through the same IAM role. Sales analysts split into EMEA, APAC, and AMER cohorts, and each cohort should see only their region’s rows, a region = 'EMEA'-style predicate applied silently by the platform, not by the query author. And every read must land in a durable log the risk team controls: who asked, which table, which columns, what filter applied, what time, which engine.

On top of all that, the four domains have classification levels (public, internal, restricted, pii), and the team is tired of rewriting a dozen principal-to-resource grants every time a table lands.

What actually matters

Before reaching for a service, it’s worth weighing what is actually broken.

The unit of permission is at the heart of the problem. IAM-on-S3 grants permission on an S3 object key, a path, not a schema. The policy surface knows nothing about tables, columns, or rows; it knows about prefixes. That mismatch is why the sales analyst reads the commission column. Any fix that stays at the object layer is going to fight the schema; any fix that moves the policy surface up to the catalog layer can speak the same language as the query.

The ownership of the rule matters too. Today, a regional-analyst filter is either in the analyst’s head (they promise to write WHERE region = 'EMEA') or in a per-view layer of custom SQL the data team maintains. Neither is enforceable; the curious analyst can just not type the filter. The rule needs to live somewhere the engine consults whether or not the query author cooperates.

The blast radius of a misstep deserves explicit thought. When the commission column is protected by “the sales analyst’s role doesn’t have access to the view that includes it,” a typo in the view name, a forgotten GRANT, or a direct query against the base table all leak the column. When the column is protected by a filter the engine applies before results leave the worker, there is no base-table query that can route around it, the rule is structural, not conventional.

The cost shape of the policy model is the other side. Named-resource grants scale as principals x resources: forty tables times eight analyst groups is 320 grants, growing every time a table or a cohort does. Tag-based grants scale as principals + resources: attach a tag to the table once, grant principals against the tag once. The difference between those two growth curves is the difference between a policy model the team will maintain and one they will abandon.

The audit requirement adds another dimension. “Every query Alice ran against sales.transactions in Q3, with the filter applied” needs a single log the risk team consumes, not three engines’ query histories stitched together by hand. Whatever the fix is, it has to emit a unified event per access, correlatable across engines.

And finally, the coupling to the engines. The data stays in S3, registered in the catalog, queried through several different engines. A fix that’s native to only one engine leaves the others with the original problem. The fix needs to live at a layer every engine consults before reading.

What we’ll filter on

1. Row-level security, a per-principal predicate applied to rows before results leave the engine.
2. Column-level security, columns absent from results for restricted principals, even when the file on disk contains them.
3. Tag-based policy model, permissions expressed against classification tags so a new tagged table inherits without a per-grant edit.
4. Cross-service enforcement. Athena, Redshift Spectrum, EMR (Spark), and Glue ETL all honour the same model from the same definition.
5. Centralised audit log of every query: principal, table, columns, filter applied, timestamp, engine.

The data lake access-control landscape

IAM-on-S3 prefix policies (the status quo). The unit of permission is the S3 object; there is no knowledge of table schema. Row-level filtering can be faked by duplicating data into per-region prefixes, but that fights the year=/month=/ partition layout and falls apart the moment a column-level hide is needed within one region’s prefix.

Custom views with IAM role switching. The data team writes a view per (region, role, column-set) combination, giving each group access to the view rather than the base table. Works at small scale; scales quadratically. Audit logging is whatever the engine emits for a view access, no unified trail across engines.

Redshift-native row-level security. Redshift has first-class RLS policies, granted to principals, applied transparently to SELECT. Combined with column-level privileges, it covers row-and-column requirements cleanly for data inside Redshift. The data stays in S3 as Parquet, queried through Redshift Spectrum; RLS doesn’t extend to other engines, so a Spark job on EMR reading the same Parquet files would see unfiltered rows.

Third-party governance platform (Immuta, Privacera, Okera). An overlay policy plane: write policies in one place, enforce through query-rewrite plugins installed into each engine. Technically capable. Carries per-seat licensing and a second policy system to keep in sync with the AWS-native one.

AWS Lake Formation. AWS’s dedicated data-lake access control service. The Glue Data Catalog becomes the policy surface: permissions are granted on catalog objects to IAM principals, with optional data filters that add row-level predicates and column-level include/exclude lists. Permissions can be granted by named resource or by LF-Tag expression. Athena, Redshift Spectrum, EMR (with the Lake Formation-enabled runtime), and Glue ETL ask Lake Formation at query time which rows and columns this principal may see; Lake Formation returns temporary, scoped-down credentials and a filter the engine applies before returning results.

Side by side

Option	Row-level	Column-level	Tag-based	Cross-engine	Central audit
IAM-on-S3 prefix policies	✗	✗	—	✗	—
Custom views + IAM switching	✓	✓	✗	✗	✗
Redshift-native RLS	✓	✓	✗	✗	—
Third-party governance platform	✓	✓	✓	✓	✓
AWS Lake Formation	✓	✓	✓	✓	✓

Two survivors. The third-party platform works but costs money and adds a second control plane. Lake Formation is the AWS-native answer: same bucket, same Parquet, same catalog, permissions and filters layered on top, CloudTrail as the audit sink.

Where the policy surface moves

Lake Formation, in depth

Lake Formation moves the authorisation decision from “can this IAM principal read these S3 bytes” to “can this IAM principal run this SELECT against this catalog table, and what filter applies.”

The catalog becomes the policy surface. After a bucket is registered with Lake Formation, direct IAM access to the underlying S3 objects is replaced by Lake Formation-issued temporary credentials. Integrated engines call Lake Formation’s GetTemporaryGlueTableCredentials at plan time; Lake Formation checks the principal against the catalog grants, computes any data filter that applies, and vends short-lived credentials scoped to exactly the S3 objects the engine needs to read.

LF-Tags carry the classification model. An LF-Tag is a key-value label attached to databases, tables, or columns, distinct from IAM tags or S3 object tags, a Lake Formation-only concept. Tags attached to a database cascade to its tables and columns by inheritance, overridable at any level. For this scenario the domain owners tag sales.transactions with classification=restricted, tag the commission_amount column specifically with classification=pii (overriding the table default), and a single grant of SELECT ON TABLES WITH LF-TAGS ('classification'='restricted') to the sales-analyst role covers every existing restricted table and every future one tagged the same way, the column’s pii tag overrides and excludes it from that grant automatically. A separate grant to the auditor role on classification=pii brings the column back for that role.

Named-resource grants scale as principals x resources; LF-Tag grants scale as principals + resources, which is the difference between a maintained policy and an abandoned one on a 40-table estate.

Data filters carry the row and cell rules. A data filter is a catalog object bound to one table, consisting of a row-filter expression (PartiQL WHERE-clause syntax, e.g. region = 'EMEA') and a column specification (explicit include or exclude list). A filter with column-wildcard and a row expression is row-level security; a filter with a column exclude list and no row expression is column-level security; both together is cell-level. Filters are granted as SELECT ON TABLE sales.transactions WITH DATA FILTER emea_no_commission TO ROLE sales-analyst-emea.

The engine applies the filter server-side. The analyst writes SELECT * FROM sales.transactions. Athena’s planner asks Lake Formation for the effective view; Lake Formation returns the filter; Athena rewrites the plan to include the row predicate and the column list before execution. Worker nodes apply both; the commission column never materialises in a result row for a non-auditor; an APAC row never leaves the worker for an EMEA analyst, regardless of what WHERE clause they write themselves.

Engines enforce consistently. Athena works out of the box. Redshift Spectrum supports row-level and cell-level filters, with the one gotcha that IAM role chaining is not supported. EMR requires the Lake Formation-enabled runtime; plain EMR Spark without that runtime bypasses fine-grained filtering. Glue Studio-authored jobs pick up Lake Formation natively.

CloudTrail is the audit primitive. Lake Formation emits management events (GrantPermissions, CreateDataCellsFilter) and a data event called GetDataAccess, the call an engine makes when it requests credentials for a specific query. The event records the principal’s ARN, the catalog resource, the permissions applied, and the request context. Combined with the engine’s own query log correlated on query execution ID, it produces a defensible audit story without building a new log pipeline.

Operational details worth spelling out

Registering a bucket is a mode switch, not an overlay. When a bucket is registered with Lake Formation, old IAM-based paths don’t automatically stop working. Lake Formation and IAM both evaluate, and if either allows, the bytes come out. To actually enforce catalog policy, the legacy IAM statements granting s3:GetObject on the bucket must be removed from the analyst roles, and Lake Formation’s IAMAllowedPrincipals legacy grant (enabled by default on older catalogs) must be revoked.

LF-Tags inherit, and the override is the feature. Attaching classification=restricted to a database cascades to its tables and columns; attaching classification=pii to one column overrides the table’s tag on just that column. That override is exactly what makes the commission_amount case tractable, the table is restricted (so analysts can SELECT from it), the column is pii (so only auditor-role SELECT reaches it).

CloudTrail’s GetDataAccess does not record query text. It is a permission-check event, not a query-execution event. For the full audit story. SQL text, row count, duration, the risk team correlates CloudTrail with the engine’s own query log. The join key is the engine’s query execution ID.

What’s worth remembering

1. IAM-on-S3 is the wrong layer for table-aware access control. The unit of permission is the S3 object; the requirement needs the unit to be the row and the column.
2. Lake Formation makes the Glue Data Catalog the policy surface. Permissions are granted on catalog objects to IAM principals, and integrated engines request scoped credentials at plan time rather than holding direct S3 access.
3. LF-Tags are a Lake Formation-only concept (distinct from IAM tags), attached to catalog resources with inheritance from database to table to column and explicit overrides at any level. A column’s LF-Tag overrides its table’s.
4. Named-resource grants scale as principals x resources; LF-Tag grants scale as principals + resources. For a 40-table, multi-classification estate, tag-based is the maintainable shape.
5. Data filters carry row and cell rules. Row-filter expression in PartiQL, column include- or exclude-list, granted to principals. One table can carry many filters, each for a different principal set.
6. The engine applies the filter server-side before results leave the worker. An analyst cannot out-write it in their own SQL.
7. Athena, Redshift Spectrum, EMR (with the Lake Formation-enabled runtime), and Glue ETL all honour the same permissions. Redshift Spectrum explicitly supports row-level and cell-level filters.
8. CloudTrail GetDataAccess is the audit primitive, per query, per principal, per table, per filter, correlated with engine query logs via the query execution ID.
9. Registering a bucket with Lake Formation is a mode switch. Legacy IAM S3 statements and the IAMAllowedPrincipals default grant must both be removed before catalog permissions are actually enforced.

Register the acme-lake bucket with AWS Lake Formation; model classification and domain as LF-Tags; attach tags to databases, tables, and the commission_amount column (overridden to classification=pii); define per-region data filters on sales.transactions and sales.commissions with region = 'EMEA|APAC|AMER' predicates and commission_amount excluded; grant analyst roles through the filters and tag-based SELECT; grant the auditor role a no-exclusion filter plus SELECT ON ALL COLUMNS WITH LF-TAGS ('classification'='pii'); remove the analyst roles’ direct s3:GetObject statements and revoke IAMAllowedPrincipals; rely on CloudTrail GetDataAccess events, correlated with Athena, Redshift, and EMR query logs on the query execution ID, for the audit trail. Same bucket, same Parquet, same 40 tables, different policy surface.