Script to Gather Retention Details of S3 Buckets

Introduction 

This script is designed to retrieve and display retention details for all S3 buckets in your AWS account.  The script queries each bucket for life-cycle configurations and generates a comprehensive report in JSON format. Additionally, you can format the results into a tabular structure for easy viewing. 

Purpose 

• Fetch, Identify and display retention policies for each bucket. 
• Retrieve folder-specific lifecycle policies where applicable. 
• Captures post-retention actions (e.g., expiration, transition). 
• Save results in a JSON file for further processing. 

Prerequisites 

AWS CLI must be installed and configured with appropriate permissions to list S3 buckets and view life-cycle configurations. 

Script 

#!/bin/bash 
  
# Created by: Ashley Merrin Shaji
  
# Get the list of all S3 buckets in your account 
buckets=$(aws s3api list-buckets --query "Buckets[].Name" --output text) 
  
# Create a file to store the results 
output_file="s3_bucket_lifecycle_details.json" 
echo "[" > $output_file # Start the JSON array 
  
# Loop through each bucket 
for bucket in $buckets; do 
  echo "Fetching details for bucket: $bucket" 
    
  # Get the bucket's lifecycle configuration 
  lifecycle=$(aws s3api get-bucket-lifecycle-configuration --bucket "$bucket" 2>/dev/null) 
    
  # If lifecycle configuration exists, process it 
  if [ $? -eq 0 ]; then 
    echo "{\"Bucket\": \"$bucket\", \"LifecycleConfiguration\": $lifecycle," >> $output_file 
      
    # Fetch prefix-specific lifecycle rules with actions 
    prefixes=$(echo "$lifecycle" | jq -r '.Rules[]? | { 
      Prefix: .Filter.Prefix, 
      Expiration: .Expiration, 
      Transition: .Transitions[]? // null 
    }') 
      
    if [ -n "$prefixes" ]; then 
      echo "\"PrefixDetails\": [$prefixes]}" >> $output_file 
    else 
      echo "\"PrefixDetails\": []}" >> $output_file 
    fi 
  else 
    echo "{\"Bucket\": \"$bucket\", \"LifecycleConfiguration\": null, \"PrefixDetails\": []}," >> $output_file 
  fi 
done 
  
# Close the JSON array properly 
sed -i '$ s/,$//' $output_file # Remove the trailing comma from the last entry 
echo "]" >> $output_file 
  
echo "All bucket details with lifecycle configurations, prefix-specific rules, and post-retention actions saved to $output_file" 

Command to Format and Display Output 

Use the following command to parse and display the retention information in a tabular format: 

cat s3_bucket_lifecycle_details.json | jq -r ' 
[ 
.[] | 
{ 
  "Bucket": .Bucket, 
  "PostRetentionActions": ( 
    if .LifecycleConfiguration != null and .LifecycleConfiguration.Rules != null then 
      .LifecycleConfiguration.Rules[] | 
      ("Prefix: " + (.Filter.Prefix // "No Prefix") + ", Expiration Days: " + (.Expiration.Days | tostring // "No Expiration") + ", Transition Action: " + 
      ( 
        if .Transitions != null then 
          (.Transitions[] | ("After " + (.Days | tostring) + " Days -> " + .StorageClass)) 
        else 
          "No Transition" 
        end 
      )) 
    else 
      "No Retention Policy" 
    end 
  ) 
} 
] | 
(["Bucket Name", "Post-Retention Actions"] | @tsv), 
(.[] | [.Bucket, .PostRetentionActions] | @tsv) 
' | column -t -s$'\t' 
' 

Output 

The script generates a table with the following columns: 
Bucket Name: The name of the S3 bucket. 
Retention Policy: 
The jq command extracts and formats the following details for each bucket: 

• "Prefix": Displays the prefix (folder) for which the lifecycle policy is applied. 
• "Expiration Days": Displays the number of days after which objects expire. If no expiration policy is set, it shows "No Expiration". 
• If no lifecycle configuration is present for the bucket, it displays "No Retention Policy". 

JSON Output: 

[ 
  { 
    "Bucket": "example-bucket-1", 
    "LifecycleConfiguration": { 
      "Rules": [ 
        { 
          "Filter": { "Prefix": "folder1/" }, 
          "Expiration": { "Days": 30 }, 
          "Transitions": [ 
            { "Days": 60, "StorageClass": "GLACIER" } 
          ] 
        } 
      ] 
    }, 
    "PrefixDetails": [ 
      { 
        "Prefix": "folder1/", 
        "Expiration": { "Days": 30 }, 
        "Transition": { "Days": 60, "StorageClass": "GLACIER" } 
      } 
    ] 
  }, 
  { 
    "Bucket": "example-bucket-2", 
    "LifecycleConfiguration": null, 
    "PrefixDetails": [] 
  } 
] 

Tabular Output: 

Conclusion 

This script provides a comprehensive way to retrieve and display life-cycle policies for all S3 buckets in an AWS account, including folder-specific life-cycle rules and post-retention actions like expiration and transitions to different storage classes. It generates a detailed JSON report, helping users understand how long objects are retained or when they are transitioned or deleted. 

Additionally, the provided command formats the output into a clear, tabular view, making it easy for administrators to quickly identify which buckets or folders have specific retention policies and whether objects are set to expire or transition to another storage class. 

This solution ensures efficient data life-cycle management across all S3 buckets, aiding in compliance, cost optimization, and data governance.