Six Petabytes and Nobody Knows Which Drawer

The situation

A genomics institute holds roughly six petabytes and is running out of room in its on-premises array. Five distinct kinds of data live on it.

Sequencer output under active processing. Each run produces 1.2 TB, and a pipeline reads it heavily for about seventy-two hours. Roughly twelve runs a week. After processing, the raw file is not touched again except under audit.

A shared analysis workspace. Around sixty researchers on Linux workstations and a compute cluster read and write the same directory tree simultaneously. Standard POSIX file semantics, existing tools that expect a mounted path, and no appetite for rewriting any of them.

Results served to a public portal. Around 40 TB of processed result files, downloaded by researchers worldwide, thousands of requests a day, accessed through a web application.

Twenty years of raw reads. About 5 PB, retained for twenty-five years under a research funding condition. Read perhaps twice a year, in response to a reproducibility request, and when read the researcher will wait: a day is acceptable.

A nightly scratch area. About 8 TB of intermediate files, generated and deleted every night. If a node dies mid-run the job restarts from the beginning anyway.

The institute also has a local instrument control system that writes to an SMB share and cannot be changed.

What actually matters

The first discriminator is how the data is reached, and it separates the options before size or cost enters the conversation. Some data is retrieved as whole objects over HTTP by an application that knows the key. Some has to appear as a mounted file system to tools that open, seek and append. Some has to look like a raw disk to a single machine. Those are three different storage types, and no amount of capacity planning converts one into another. Choosing object storage for a workload that needs POSIX semantics means rewriting sixty researchers’ tooling.

The second is how often the data is read, because storage classes are priced on the assumption that cheaper storage is read less. A class with a low storage price and a retrieval charge is cheaper overall only when the data is genuinely cold, and it becomes more expensive than the standard class if something scans it monthly. Five petabytes read twice a year and forty terabytes read thousands of times a day sit at opposite ends of that trade, and putting either in the other’s class is expensive in a different direction each time.

Third, how quickly it has to come back when it is read, which is a separate axis from how often. Archived data that a researcher will wait a day for is a different requirement from archived data needed in milliseconds, and the archive classes differ precisely on that.

Fourth, whether the data can be regenerated. Data that a nightly job rebuilds from scratch does not need to survive anything, and paying for durability and replication it will never use is waste. That is the one case where the ephemeral option is correct rather than dangerous.

Finally, lifecycle. A twenty-five year retention rule implemented as a calendar reminder is a rule that will be broken. Implemented as a policy on the storage itself, it becomes configuration that runs whether or not anyone remembers.

What we’ll filter on

1. How the data is reached: object over HTTP, a mounted file system, or a block device on one instance.
2. How often it is read, since a lower storage price comes with a retrieval charge that frequent reads would undo.
3. How fast it has to come back: milliseconds, minutes, or hours.
4. Whether losing it matters, or whether it can be regenerated.
5. Whether the retention rule can be expressed as a policy rather than a habit.

The landscape

Amazon S3 stores objects retrieved by key over HTTP, with effectively unlimited capacity and eleven nines of durability. It fits whenever a whole file is written once and read as a unit, and not when a tool expects to mount a path. Most of the cost decision is the choice of storage class.

S3 storage classes run from hot to cold, and each cold one sets a minimum storage duration you are billed for even if you delete early. S3 Standard is for frequently accessed data, with no minimum. S3 Intelligent-Tiering moves objects between access tiers automatically for a per-object monitoring and automation charge, with no retrieval fees, and suits an access pattern nobody can predict. S3 Standard-IA has a lower storage price, a per-GB retrieval charge and a thirty-day minimum, for data needed quickly but rarely. S3 One Zone-IA is the same but stored in a single Availability Zone, cheaper and correct only for reproducible data. S3 Glacier Instant Retrieval is archive pricing with millisecond access and a ninety-day minimum. S3 Glacier Flexible Retrieval also has a ninety-day minimum, and a restore takes 1 to 5 minutes expedited, 3 to 5 hours standard, or 5 to 12 hours bulk. S3 Glacier Deep Archive has the lowest storage price of any class and a one-hundred-and-eighty-day minimum; a standard restore finishes within 12 hours and a bulk one within 48.

Amazon EBS is a block volume attached to one EC2 instance, behaving like a disk and persisting independently of the instance. It suits a boot volume or a database’s data directory. It attaches to one instance at a time in the ordinary case, which makes it unsuitable for sixty researchers sharing a workspace.

EC2 instance store is block storage on disks physically attached to the host, and it carries no separate charge because it is included in the instance price. It is ephemeral: the data survives a reboot, and is lost when the instance is stopped, hibernated or terminated. That is a disqualifying property for anything that matters and the right property for scratch.

Amazon EFS is a managed NFSv4 file system that many Linux instances mount at once. A Regional file system stores the data across several Availability Zones, and capacity grows and shrinks with the files rather than being provisioned. It is the answer to a shared POSIX workspace. It has Standard, Infrequent Access and Archive storage classes, with lifecycle policies moving files between them. Windows instances cannot mount it.

Amazon FSx provides managed versions of file systems that already exist elsewhere. FSx for Windows File Server speaks SMB versions 2.0 to 3.1.1 and authenticates users against Active Directory. FSx for Lustre is a POSIX-compliant parallel file system for compute-intensive Linux work, and it links to an S3 bucket. There are NetApp ONTAP and OpenZFS variants too.

AWS Storage Gateway gives on-premises systems access to AWS storage through a local cache. Amazon S3 File Gateway presents NFS v3 or v4.1 and SMB v2 or v3, storing the files as S3 objects. Amazon FSx File Gateway presents SMB shares backed by FSx for Windows File Server. Volume Gateway presents iSCSI block volumes, and Tape Gateway presents a virtual tape library to existing backup software.

AWS Backup centralises backup policy across EBS, RDS, DynamoDB, EFS, FSx, S3 and others, with schedules, retention rules, cross-Region copies and compliance reporting in one place.

Bulk transfer is its own problem at six petabytes. AWS DataSync moves data online from NFS, SMB, HDFS and self-managed object storage into S3, EFS or FSx, and a single task can saturate a 10 Gbps link. The AWS Snow Family devices that used to cover the offline case are closed to new customers, so don’t reach for them. The offline alternative AWS names is AWS Data Transfer Terminal: a physical facility you book a slot at and bring your own drives to.

Evaluation

Side by side

Option	Access method	Suits frequent reads	Suits rare reads	Survives instance loss	Shared by many hosts
S3 Standard	Object over HTTP	✓	✗	✓	✓
S3 Standard-IA	Object over HTTP	✗	✓	✓	✓
S3 Glacier Instant Retrieval	Object over HTTP	✗	✓	✓	✓
S3 Glacier Deep Archive	Object, hours to restore	✗	✓	✓	✓
Amazon EBS	Block, one instance	✓	✓	✓	✗
Instance store	Block, ephemeral	✓	✗	✗	✗
Amazon EFS	NFS, many instances	✓	✓	✓	✓
FSx for Windows File Server	SMB, many hosts	✓	✓	✓	✓
FSx for Lustre	Parallel file system	✓	✗	✓	✓
Storage Gateway	NFS, SMB or iSCSI on-premises	✓	✓	✓	✓

The access-method column does most of the eliminating. Everything else is a cost and speed trade inside the family the access method has already chosen.

Where each dataset lands

Dataset	Access pattern	Storage	Why
Sequencer output in processing	Heavy parallel reads for 72 hours	FSx for Lustre, linked to S3	Parallel throughput for the pipeline, backed by the bucket
Sequencer output after processing	Untouched except under audit	S3 Glacier Flexible Retrieval	Read perhaps never, and a wait is acceptable
Shared analysis workspace	60 researchers, POSIX, concurrent	Amazon EFS	The only option giving a mounted shared file system to Linux hosts
Public portal results	Thousands of reads a day	S3 Standard, behind CloudFront	Frequently read, served over HTTP, cached at the edge
Twenty years of raw reads	Twice a year, a day’s wait acceptable	S3 Glacier Deep Archive	Lowest storage price; a standard restore finishes within 12 hours
Nightly scratch	Rebuilt every night	Instance store	Regenerated anyway, and billed with the instance rather than separately
Instrument control SMB share	Unchangeable local system	Amazon S3 File Gateway	Presents SMB locally, stores objects in S3

The solution

Start with the five petabytes, because it is most of the estate. Twenty years of raw reads, read twice a year, with a day’s wait acceptable, is S3 Glacier Deep Archive: the lowest storage price of any class, with a standard restore finishing within 12 hours. Ask for standard rather than bulk retrieval, because bulk takes up to 48 hours and misses the requirement. Express the twenty-five year retention as a lifecycle policy on the bucket rather than as an operational habit. Object Lock in compliance mode covers the case where the funding condition says the data cannot be deleted early. It needs S3 Versioning on the bucket. Once a version is locked, no user, not even the account root user, can delete it or shorten its retention period. Getting six petabytes there over an institutional link is its own project. Work out the transfer time before assuming an upload; DataSync over a Direct Connect hosted connection is the route AWS now points at.

The public portal results go to S3 Standard with CloudFront in front. Forty terabytes read thousands of times a day is frequently accessed data by any definition, and putting it in an infrequent-access class would add a retrieval charge to every one of those reads. CloudFront caches the popular files at edge locations, which improves download speed for researchers on other continents and reduces the request volume reaching the bucket.

The shared analysis workspace goes to EFS. Sixty researchers and a compute cluster reading and writing one directory tree at the same time, with tools that expect a mounted path, is precisely what a managed NFS file system is for. It is the one requirement here that object storage cannot satisfy at all. A shared workspace accumulates, so enable lifecycle policies: files nobody has touched for thirty days move to Infrequent Access, and files untouched for ninety move to Archive.

Active sequencer runs go to FSx for Lustre, linked to the S3 bucket. The pipeline reads 1.2 TB heavily for seventy-two hours, which is the compute-intensive parallel access Lustre exists for, and the S3 link means the file system is populated from the bucket and results are written back to it. When the run finishes, the file system can be deleted and the data lives in S3, transitioning to Glacier Flexible Retrieval on a lifecycle rule.

The nightly scratch goes on instance store, and this is the one case where ephemeral is correct rather than reckless. The files are regenerated every night, a failed job restarts from the beginning regardless, and instance store volumes sit on disks attached to the host with no charge beyond the instance itself. Durable, replicated storage for data that will not exist in the morning is the mistake to avoid here.

The instrument control system gets an Amazon S3 File Gateway. It keeps writing to an SMB share exactly as it does now, with a local cache for recent files, and the data lands in S3 as objects where the rest of the pipeline can reach it. Nothing about the instrument software changes, which was the constraint.

Finally, put AWS Backup over the parts that need it. EFS and the FSx file systems get backup plans with retention rules and cross-Region copies where the funding condition requires geographic separation, and compliance reporting comes from one place rather than from three consoles. AWS Backup covers S3 too, but not objects in Glacier Flexible Retrieval or Deep Archive, so versioning and Object Lock are what protect the archive.

What’s worth remembering

1. Choose the storage type by how the data is reached: object over HTTP is S3, a mounted shared file system is EFS or FSx, and a disk attached to one instance is EBS.
2. Instance store data survives a reboot but is lost when the instance stops, hibernates or terminates, which makes it correct for regenerated scratch data and wrong for anything else.
3. S3 classes trade storage price against retrieval charge and retrieval time: Standard for frequent reads, Standard-IA for rare reads needed quickly, Glacier Instant Retrieval for archive needed in milliseconds, Flexible Retrieval for minutes to hours, Deep Archive for the lowest price and a wait of up to 12 hours.
4. The cold classes each bill a minimum storage duration whether or not you keep the object that long: thirty days for the IA classes, ninety for the two cheaper Glacier ones, one hundred and eighty for Deep Archive.
5. Intelligent-Tiering suits an access pattern nobody can predict; it moves objects between tiers automatically for a per-object monitoring charge and charges no retrieval fee.
6. Lifecycle policies and Object Lock turn a retention rule into configuration, and AWS Backup centralises backup schedules, retention and compliance reporting across EBS, RDS, DynamoDB, EFS and FSx.