Imagine you're in a security assessment with your Compliance Team or an External Assessor, and they point out that your AWS environment CloudWatch Log doesn't have a retention period set.
CloudWatch Logs retention is an important part of AWS security compliance that is often missed. When log groups are set to "Never Expire," organizations face several major risks:
The Risks of Unlimited Log Retention
Uncontrolled Cost Growth: With no expiration policy, logs accumulate indefinitely, leading to steadily increasing storage costs that can spiral out of control over time.
Audit Non-Compliance: Many compliance frameworks require specific data retention periods, neither too short nor too long. Indefinite retention may violate regulatory requirements for data deletion.
SIEM Data Volume Overload: Security Information and Event Management systems can become overwhelmed with excessive historical data, potentially degrading performance and increasing processing costs.
Now that we understand the risks of logs that never expire and how they violate compliance, there is usually a recommendation after the assessment on how to fix this according to best practices.
It's important to know that different environments have different framework controls and security best practices. Some environments might have a Record Retention Policy set to (Calendar Year +1), while others might be more than 90 days. In this lab, we will set it to 365 days of retention.
Implementing Automated Log Retention
Let's explore how to implement proper CloudWatch Log retention on existing logs using both AWS CLI and a Python boto3 Lambda function.
Option 1: AWS CLI Remediation
You can use the AWS CLI to identify log groups without retention policies and set appropriate retention periods:
aws logs describe-log-groups \
--query "logGroups[?retentionInDays==null].logGroupName" \
--output text
If you have JQ installed on your local machine, you can use the AWS CLI query below instead of the one above.
aws logs describe-log-groups --query "logGroups[?retentionInDays==null].logGroupName" | jq
The screenshots above show a list of CloudWatch Logs without a 365-day retention period. Now, we will select one of the logs and run the next CLI command to change it to 365 days.
aws logs put-retention-policy --log-group-name "/aws-athena/Spark_primary" --retention-in-days 365
Option 2: Lambda Function with Boto3
For a more automated approach, we can create a Lambda function that runs on a schedule. In this lab, we'll use a simple execution Lambda, but I will provide a GitHub repository with the automated method.
Go to your AWS console, navigate to Lambda functions, and create a function using the Python runtime. Set your IAM role with permissions for CloudWatch. Paste the code below into your Lambda function and execute it.
import boto3
def lambda_handler(event, context):
client = boto3.client('logs')
paginator = client.get_paginator('describe_log_groups')
for page in paginator.paginate():
for lg in page['logGroups']:
if lg.get('retentionInDays') is None:
client.put_retention_policy(
logGroupName=lg['logGroupName'],
retentionInDays=365
)
print(f" Set 365-day retention for {lg['logGroupName']}")
Below is the output from the executed Lambda function:
Conclusion
Implementing proper CloudWatch Log retention is a crucial security and cost management practice. By setting appropriate retention periods:
You gain predictable and controlled costs
You maintain compliance with audit requirements
You prevent SIEM systems from being overwhelmed with excessive data
