We recommend applying a lifecycle policy (including Intelligent Tiering) for S3 buckets to limit costs. Act with caution and ensure you implement the correct lifecycle policies.
Subtitles are available in English, French and Spanish. Click here to get the full masterclass.
How to identify?
You can identify S3 buckets that need lifecycle policies using the following methods:
From our Cost Optimizations Report, check at the S3_Storage_Classes section. The buckets listed are over 1 TB and primarily use the most expensive Standard tier.
In AWS Console, go to the S3 service, then select the Buckets section. Click on "View Storage Lens dashboard" at the top.
Go to the Top N overview at the bottom and modify the number to 10.
Review the list of your buckets, their total storage, and trends.
Every S3 bucket over 1 TB should be investigated.
In AWS Console, go to the Billing service, then select the Bills section. Choose Simple Storage Service and a region. Search for the requests line.
How to fix?
In AWS Console, go to the S3 service, then select the Buckets section.
Click on a bucket name.
Recognize the data in each folder and filter by the Last modified column.
Go to Properties to check if Bucket Versioning is enabled.
Go to Metrics to see the trend of your bucket (bucket size and number of objects).
Go to Management to review existing Lifecycle rules and Replication rules.
Ask yourself the following questions:
1. Do I need that data ?
If the bucket is totally or partially useless, deletion is necessary.
Total Deletion :
To delete an entire bucket, empty it first. On the Buckets list, select the bucket, click Empty, and confirm.
Return to the Buckets list, select the bucket, click Delete, and confirm.
If deletion takes too long due to many objects, create a Lifecycle Policy:
Click the bucket name, go to Management, and click Create lifecycle rule.
Configure the rule as shown below and click Create rule.
Deletion will take 1-2 days. Monitor progress in the Metrics section.
Partial Deletion :
To delete specific folders or objects, select them and click Delete.
If deletion takes too long, create a Lifecycle Policy with a prefix (e.g., logs/2012/).
This example will modify all data in the folder 2012 in the folder logs of your bucket. Please create one Lifecycle policy per prefix. But one prefix can be used to delete multiple folders. If you put logs/201 in prefix, it should delete logs from 2010 to 2019 (every folder starting by 201 in folder logs).
If you just want to delete all others versions of objects, you can also add 1 in Number of newer versions to retain.
You can also automate these policies through CloudFormation, Terraform, etc.
It should take 1-2 days to delete all objects. You should see the progress in Metrics section.
2. If I need that data, then how long ?
Determine the data retention period with your team (e.g., 10 days, 3 months, 3 years). For example, logs bucket must have an expiring policy that will depends on your data retention policy. Could be 10 days, 3 months, 3 years ?
Create a Lifecycle Policy and adjust the Days section accordingly. For 1 year retention, you should put that :
3. If I need permanently that data, then how frequently do I need to access it ?
If retaining data for archival purposes, consider storage classes based on access frequency.
Please only archive objects that you are not supposed to retrieve frequently, otherwise it will be more expensive and you might need to wait to get your data.
You have many different classes, but the classic transition would be this :
Transition objects to cost-effective storage classes, ensuring a minimum object size to avoid unnecessary transition costs.
Apply the same transitions to noncurrent versions.
If unsure about data usage, use Intelligent-Tiering to automatically move objects to the correct storage type.
S3 Storage Classes
S3 provides different levels of reliability and availability through its storage classes.
Each storage class has a different cost structure and guarantees about how quickly you can access your data. Some of them even have minimal storage duration.
S3 Standard: Designed for frequently accessed data.
S3 Intelligent-Tiering: Data with changing or unknown access patterns.
S3 Standard-IA: Designed for long-lived, infrequently accessed data.
S3 One Zone-IA: Designed for long-lived, infrequently accessed, non-critical data.
S3 Reduced redundancy: Frequently accessed, non-critical data.
S3 Glacier Instant Retrieval: Designed for archive data accessed once a quarter with instant retrieval in milliseconds.
S3 Glacier Flexible Retrieval (formerly Glacier): Designed for infrequently accessed, archived critical data.
S3 Glacier Deep Archive: Long-term data archiving with retrieval times within 12 hours.
You can also analyze all your bucket with S3 analytics.
If the resource is provisioned automatically via IaC (such as CloudFormation or Terraform), update the corresponding code to reflect the changes.