AWS Storage Best Practices: S3, EBS, and EFS Architecture Guide

Technical TL;DR

Storage costs grow 30-50% annually without governance. Key takeaways:

---

1. S3 (Simple Storage Service) Best Practices

1.1 Choose the Right Storage Class

S3 has 7+ storage classes. Using the wrong one wastes money.

| Storage Class | Availability | Durability | Use Case | Cost vs Standard |

|---------------|--------------|------------|----------|------------------|

| **Standard** | 99.99% | 99.999999999% | Frequently accessed data | Baseline |

| **Intelligent-Tiering** | 99.9% | 99.999999999% | Unknown access patterns | +$0.0025/GB monitoring |

| **Standard-IA** | 99.9% | 99.999999999% | Infrequent access (30+ days) | ~50% cheaper |

| **One Zone-IA** | 99.5% | 99.999999999% | Non-critical, reproducible data | ~60% cheaper |

| **Glacier Instant Retrieval** | 99.9% | 99.999999999% | Quarterly access (1-5 min retrieval) | ~65% cheaper |

| **Glacier Flexible Retrieval** | 99.99% | 99.999999999% | Annual access (minutes-hours retrieval) | ~85% cheaper |

| **Glacier Deep Archive** | 99.99% | 99.999999999% | 10+ year retention (12-hour retrieval) | ~95% cheaper |

Selection Framework:

```yaml

Access Frequency → Storage Class

─────────────────────────────────────────

Daily+ → S3 Standard

Weekly-Monthly → Standard-IA (after 30 days)

Quarterly → Glacier Instant Retrieval

Annually → Glacier Flexible Retrieval

10+ Year Retention → Glacier Deep Archive

Unknown Patterns → Intelligent-Tiering (auto-moves)

```

1.2 S3 Lifecycle Policies: Automate Tiering

Lifecycle policies automatically move data to cheaper tiers.

Required Policy for Production Buckets:

```yaml

Lifecycle Rule: Standard → IA → Glacier Deep Archive

───────────────────────────────────────────────────

Transition Object (after 30 days) → Standard-IA

Transition Object (after 90 days) → Glacier Flexible Retrieval

Transition Object (after 180 days) → Glacier Deep Archive

Expiration Object (after 7 years) → Delete (compliance)

```

Implementation:

```json

{

"Rules": [

{

"Id": "MoveToIA",

"Status": "Enabled",

"Transitions": [

{ "Days": 30, "StorageClass": "STANDARD_IA" },

{ "Days": 90, "StorageClass": "GLACIER" },

{ "Days": 180, "StorageClass": "DEEP_ARCHIVE" }

],

"Expiration": { "Days": 2555 } // 7 years

}

]

}

```

Cost Impact: Typical S3 bill reduced by 40-60% with lifecycle policies.

1.3 S3 Security: Public Access Blocked by Default

Public S3 buckets are a top security finding. Block public access.

S3 Bucket Security Checklist:

```yaml

☐ Block Public Access: ON (all 4 settings)

☐ Bucket Policy: Deny all non-HTTPS (s3:x-amz-server-side-encryption)

☐ Server-Side Encryption: AES-256 or aws:kms

☐ Versioning: Enabled (critical data protection)

☐ MFA Delete: Enabled (prevent accidental deletion)

☐ Access Logs: Delivered to S3 (separate account)

☐ Object Lock: WORM compliance (legal/regulated data)

```

Bucket Policy: Force Encryption

```json

{

"Version": "2012-10-17",

"Statement": [{

"Sid": "DenyUnencryptedObjectUploads",

"Effect": "Deny",

"Principal": "*",

"Action": "s3:PutObject",

"Resource": "arn:aws:s3:::my-bucket/*",

"Condition": {

"StringNotEquals": {

"s3:x-amz-server-side-encryption": "AES256"

}

}

}]

}

```

1.4 S3 Replication: Disaster Recovery

Cross-region replication (CRR) protects against regional disasters.

Replication Use Cases:

```yaml

☐ Disaster Recovery (RPO: minutes, RTO: hours)

☐ Compliance (data residency requirements)

☐ Latency reduction (global access)

☐ Account-level aggregation (centralized logging)

```

Configuration:

```yaml

Source Bucket (us-east-1)

↓ Replication

Destination Bucket (us-west-2)

↓

Versioning Required: YES

Encryption: SSE-KMS (replicate KMS keys)

Replication Time Control (RTC): 15 minutes (SLA-backed)

```

1.5 S3 Performance Optimization

Single object requests can achieve 3,500 PUT/5,500 GET requests per second.

For higher performance:

```yaml

☐ Use random key prefixes (hash-based partitioning)

Bad: bucket/file-001, bucket/file-002 (sequential)

Good: bucket/a1b2/file-001, bucket/c3d4/file-002 (random)

☐ Use S3 Transfer Acceleration for long-distance uploads

Cost: Additional $0.004/GB transferred

☐ Use multipart upload for files > 100MB

Parallel uploads, faster failure recovery

☐ Use S3 Byte-Range Fetches for partial object access

Skip downloading entire object

```

1.6 S3 Select: Query Data Without Downloading

S3 Select filters data at rest. Avoid full object download.

Use Cases:

Cost Benefit: 50-80% cheaper than downloading entire object.

---

2. EBS (Elastic Block Store) Best Practices

2.1 Choose the Right EBS Volume Type

EBS has 7 volume types. Wrong choice = 3-5x overspending.

| Volume Type | Use Case | Max IOPS | Cost/GB |

|-------------|----------|----------|---------|

| **gp2/gp3** (General Purpose SSD) | Boot volumes, dev/test | 16,000 (gp3) | $0.08/GB |

| **io1/io2** (Provisioned IOPS SSD) | Databases, low-latency | 64,000 | $0.125/GB + IOPS |

| **st1** (Throughput-Optimized HDD) | Big data, data warehousing | 500 MB/s | $0.045/GB |

| **sc1** (Cold HDD) | File servers, log archives | 250 MB/s | $0.025/GB |

Recommendation:

```yaml

Production → gp3 (baseline), io2 for databases

Development → gp2 (legacy), gp3 preferred

Big Data → st1 (throughput-optimized)

Archival → sc1 (lowest cost, use S3 instead)

```

2.2 gp3: The New Default

gp3 offers better price-performance than gp2.

gp3 Advantages:

```yaml

☐ Consistent performance (independent of volume size)

☐ Lower cost: $0.08/GB (vs gp2's $0.10/GB)

☐ Provisioned IOPS: 3,000-16,000 (gp2: based on size)

☐ Provisioned throughput: 125-1,000 MB/s

☐ Better cost predictability

```

Migration:

```bash

# Create snapshot of gp2 volume

# Create gp3 volume from snapshot

# Attach to instance (update /etc/fstab if needed)

# Verify and retire gp2 volume

```

2.3 EBS Encryption: Non-Negotiable

All EBS volumes must be encrypted.

Configuration:

```yaml

☐ Enable encryption at volume creation

☐ Use customer-managed KMS keys (CMKs)

☐ Separate keys per environment (dev/staging/prod)

☐ Snapshot encryption: Inherited from volume

```

2.4 EBS Snapshots: Backup Strategy

Snapshots are incremental. First snapshot = full size, subsequent = changes.

Snapshot Strategy:

```yaml

Production Volumes:

☐ Automated snapshots (AWS Backup or custom Lambda)

☐ Retention policy (daily: 7 days, weekly: 4 weeks, monthly: 12 months)

☐ Cross-region copy for DR

☐ Test restoration monthly

Snapshot Lifecycle:

☐ Transition older snapshots to S3 Glacier (cost savings)

☐ Tag snapshots with retention requirements

☐ Delete expired snapshots (prevent waste)

```

Cost Optimization:

```yaml

Snapshot Cost = Changed data since last snapshot

→ Snapshots grow slowly for unchanged data

→ Delete unused snapshots (major waste area)

→ Use AWS Backup for automated retention management

```

2.5 EBS Performance: IOPS Provisioning

Databases require provisioned IOPS. General-purpose apps use gp3.

When to Use Provisioned IOPS (io1/io2):

```yaml

✓ Production databases (MySQL, PostgreSQL, Oracle)

✓ NoSQL databases (Cassandra, MongoDB)

✓ Low-latency applications (< 5ms required)

✓ Consistent performance under variable load

```

Provisioning Guidelines:

```yaml

Database IOPS = (Transactions per second × IOPS per transaction)

Example: 1,000 TPS × 3 IOPS/transaction = 3,000 IOPS minimum

Add 20% headroom: 3,600 IOPS provisioned

```

2.6 EBS Multi-Attach: High Availability

Multi-Attach allows up to 16 EC2 instances to attach same io2 volume.

Use Cases:

```yaml

☐ Clustered Linux applications (Oracle RAC)

☐ Shared access to consistent data (not a file system!)

☐ Active-active failover (requires cluster-aware software)

```

Warning: Most applications require a cluster file system (e.g., GFS2) for multi-attach.

---

3. EFS (Elastic File System) Best Practices

3.1 When to Use EFS vs. EBS vs. S3

| Use Case | Best Solution | Why |

|----------|---------------|-----|

| **Single EC2 needs block storage** | EBS | Low cost, direct attached |

| **Multiple EC2 need shared files** | EFS | Network-attached, concurrent access |

| **Object storage, web hosting** | S3 | API access, CDN integration |

| **Container storage** | EFS or EFS CSI Driver | Shared across containers/pods |

| **Home directories** | EFS | POSIX-compliant, concurrent users |

3.2 EFS Performance Tiers

EFS offers 2 performance modes and 2 throughput modes.

Performance Modes:

```yaml

General Purpose → Recommended for most use cases

Max I/O → Required for > 7,000 file system operations/second

```

Throughput Modes:

```yaml

Bursting Throughput:

- Included with EFS (no extra cost)

- Credits accumulate when not in use

- Best for variable workloads

Provisioned Throughput:

- Pay for consistent throughput

- Required for sustained high throughput

- Cost: $0.30/MB/s/month

```

3.3 EFS Lifecycle Policies: IA for Infrequent Access

EFS Infrequent Access (EFS IA) costs 85% less than Standard.

Lifecycle Policy:

```yaml

Transition files not accessed for 30 days → EFS IA

Transition files not accessed for 90 days → EFS Archive (if enabled)

```

Cost Impact:

```yaml

EFS Standard: $0.30/GB/month

EFS IA: $0.025/GB/month (85% savings)

EFS Archive: $0.015/GB/month (95% savings)

Best For: File systems with > 20% infrequently accessed data

```

3.4 EFS Encryption: Required

Enable encryption at file system creation.

Configuration:

```yaml

☐ Encryption at rest: KMS CMK

☐ Encryption in transit: TLS (mount helper)

☐ POSIX permissions: Restrict access

☐ IAM policies: Control file system access

```

---

4. Storage Gateway: Hybrid Cloud

4.1 Storage Gateway Types

Storage Gateway connects on-premises to AWS storage.

| Gateway Type | Use Case | AWS Storage |

|--------------|----------|-------------|

| **File Gateway** | File shares, NFS/SMB | S3 |

| **Volume Gateway** | Block storage, iSCSI | S3 |

| **Tape Gateway** | Archive backup | S3 Glacier |

Use Cases:

```yaml

☐ Cloud migration (lift and shift)

☐ Data retention (on-prem + cloud backup)

☐ Low-latency local cache (hot data on-prem, cold in cloud)

☐ Disaster recovery (replicate to AWS)

```

---

5. Data Transfer and Migration

5.1 Data Transfer Cost Optimization

Data transfer OUT from AWS is expensive. IN is free.

Cost Reduction Strategies:

```yaml

1. Use CloudFront for content delivery (lower transfer costs)

2. Use VPC endpoints for AWS service access (no internet)

3. Use Direct Connect for high-volume transfers (fixed monthly cost)

4. Use Snowball for petabyte-scale one-time migrations

```

5.2 Data Migration Strategies

| Data Volume | Recommended Method |

|-------------|-------------------|

| **< 1 TB** | AWS CLI / SDK |

| **1 TB - 10 TB** | AWS DataSync (automated, scheduled) |

| **10 TB - 80 TB** | Snowball Edge (physical device) |

| **> 80 TB** | Snowball (multiple devices) |

| **Ongoing sync** | DataSync (continuous replication) |

---

6. Cost Optimization Checklist

Immediate Actions (Week 1)

Short-Term (Month 1)

Long-Term (Quarter 1)

---

7. Monitoring and Alerting

Required Metrics

```yaml

S3:

- Bucket size (AWS Trusted Advisor)

- Object count (AWS Trusted Advisor)

- 4xx/5xx error rates (CloudWatch)

- Lifecycle policy transitions

EBS:

- Volume queue length (performance bottleneck)

- Burst balance (gp2/gp3)

- Throughput (MB/s)

- IOPS utilization

EFS:

- Percent I/O limit (CloudWatch)

- Burst credit balance

- Data read/write (metered data)

```

Alerting

```yaml

☐ S3: Public access granted (EventBridge + SNS)

☐ EBS: Burst balance < 20% (performance risk)

☐ EFS: Burst credits < 20% (throttling risk)

☐ Snapshot age: No snapshot in 30 days (backup gaps)

```

---

Summary: Storage Excellence Pillars

1. **Lifecycle policies automate tiering** (reduce costs 40-60%)

2. **Encrypt everything** (S3 SSE-S3/KMS, EBS encryption, EFS KMS)

3. **Versioning for critical data** (protection against deletion)

4. **Cross-region replication for DR** (RPO minutes, RTO hours)

5. **Right-size EBS** (gp3 default, io2 for databases)

6. **Use EFS for shared access** (EBS is single-instance only)

7. **Monitor and prune snapshots** (major waste area)

8. **Block public S3 access** (security non-negotiable)

---

Need Help Optimizing Storage Costs?

Our storage architects can analyze your S3/EBS/EFS usage, implement lifecycle policies, and reduce storage costs by 40% or more.

<a href="/contact" className="text-aws-orange font-semibold hover:text-aws-light">

Schedule a Free Storage Assessment →

</a>

Internal Linking Strategy:

---

*Last updated: January 5, 2025*