Project 13: Simplify AWS Cost Management: A Step-by-Step Guide to Automated Reports
Introduction:
Uncover the secrets to effortless AWS financial management with our detailed guide, "Simplify AWS Cost Management: A Step-by-Step Guide to Automated Reports" Learn step-by-step how to set up a robust system for daily and monthly cost analyses, ensuring you gain valuable insights into your cloud expenses. Perfect for both beginners and seasoned AWS users, our guide simplifies the complexities of cost management, empowering you to optimize spending and maintain financial control over your AWS resources. Dive into the world of automated reports and discover a streamlined approach to understanding and managing your AWS costs. Read on to master your cloud finances with ease!
Technologies Used in This Project:
Amazon Simple Email Service (SES): Utilized for reliable email communication, SES played a pivotal role in sending and receiving cost reports, fostering seamless information exchange.
AWS Lambda: The serverless computing powerhouse, Lambda, was employed to execute code in response to AWS Cost Explorer API requests, automating the generation of cost reports.
AWS Identity and Access Management (IAM): IAM was instrumental in defining roles and permissions, ensuring secure access to AWS services and resources for the Lambda function.
AWS Cost Explorer API: Leveraged to gather comprehensive insights into AWS costs and usage, the Cost Explorer API facilitated the extraction of detailed billing data for analysis.
Amazon EventBridge: Used to schedule and automate the periodic execution of the Lambda function, EventBridge provided a time-based trigger for generating daily and monthly cost reports.
Python: The coding language of choice, Python, enabled the implementation of logic for fetching cost data, formatting reports, and interacting with AWS services.
Project Overview:
Project:
Step 1: Amazon SES Setup
Sender SES Setup:
- Go to Amazon SES -> Verified Identities.
Click on "Create an identity."
Choose "Email address" as the identity type, add your email, and click "Create identity."
Receiver SES Setup:
- Repeat the process to create SES for the receiver.
Verification:
- Check your email for the verification link.
- Activate your ability to receive or send emails.
- Ensure the identity status is "Verified."
Step 2: Lambda Function Setup
Create Lambda Function:
- Navigate to Lambda and click on "Create function."
- Choose runtime as Python3.9 and create a new role with basic Lambda permissions.
IAM Role Permissions:
- Add necessary permissions to the Lambda function's IAM role.
Required permissions: Cost-Explorer-GetUsage and AmazonSESFullAccess.
Below inline policy for ce-getusage
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "ce:GetCostAndUsage",
"Resource": "*"
}
]
}
Step 3: Library Package Setup
Add Library Packages as Layer:
Click on "Layers" in the Lambda function setup.
Add a layer with required library packages (boto3, DateTime, SDKPandas).
Now add Custom Datetime package
#Code you must run in local terminal to install and zip datetime library to upload mkdir custome-datetime cd custome-datetime/ pip install datetime -t . mkdir -p layer/python/lib/python3.9/site-packages/ mv DateTime/DateTime.py layer/python/lib/python3.9/site-packages/ zip -r DateTime.zip layer/
Now Click on "Layers" in the Lambda function setup. choose custom layer and select just created layer.
Step 4: Custom Python Code
Integrate Python Code:
Add Python code to the Lambda function to fetch Cost Usage from CostExplorer.
Update email addresses in the code where necessary.(You need to update at 2 places (Source and Destination).
import json
import boto3
import datetime
from botocore.exceptions import ClientError
import pandas as pd
def lambda_handler(event, context):
billing_client = boto3.client('ce')
# getting dates (yyyy-MM-dd) and converting to string
today = datetime.date.today()
yesterday = today - datetime.timedelta(days = 1)
str_today = str(today)
str_yesterday = str(yesterday)
# get total cost for the previous day
response_total = billing_client.get_cost_and_usage(
TimePeriod={
'Start': str_yesterday,
'End': str_today,
#'Start': "2023-05-16",
# 'End': "2023-05-17"
},
Granularity='DAILY',
Metrics=[ 'UnblendedCost',]
)
total_cost = response_total["ResultsByTime"][0]['Total']['UnblendedCost']['Amount']
print(total_cost)
total_cost=float(total_cost)
total_cost=round(total_cost, 3)
total_cost = '$' + str(total_cost)
# print the total cost
print('Total cost for yesterday: ' + total_cost)
# get detailed billing for individual resources
response_detail = billing_client.get_cost_and_usage(
TimePeriod={
'Start': str_yesterday,
'End': str_today,
},
Granularity='DAILY',
Metrics=['UnblendedCost'],
GroupBy=[
{
'Type': 'DIMENSION',
'Key': 'SERVICE'
},
{
'Type': 'DIMENSION',
'Key': 'USAGE_TYPE'
}
]
)
resources = {'Service':[],'Usage Type':[],'Cost':[]}
for result in response_detail['ResultsByTime'][0]['Groups']:
group_key = result['Keys']
service = group_key[0]
usage_type = group_key[1]
cost = result['Metrics']['UnblendedCost']['Amount']
cost=float(cost)
cost=round(cost, 3)
if cost > 0:
cost = '$' + str(cost)
resources['Service'].append(service)
resources['Usage Type'].append(usage_type)
resources['Cost'].append(cost)
df = pd.DataFrame(resources)
html_table = df.to_html(index=False)
print(resources)
message = 'Cost of AWS training account for yesterday was'
html = """
<html>
<head>
<style>
body {{
font-family: Arial, sans-serif;
color: white;
background-color: black;
}}
h2 {{
color: white;
font-size: 25px;
text-align: center;
}}
h1 {{
color: #333333;
font-size: 40px;
text-align: center;
background-color: yellow;
}}
p {{
color: white;
font-size: 30px;
line-height: 1.5;
margin-bottom: 20px;
text-align: center;
}}
p1 {{
font-size: 10px;
text-align: center;
margin-left: auto;
margin-right: auto;
}}
</style>
</head>
<body>
<p> Training Account report for the day {} </p>
<h2> {} </h2>
<h1> <strong> <em> {} </em></strong> </h1>
<p1>{}</p1>
</body>
</html>
""".format(str_yesterday,message,total_cost,html_table)
ses_client = boto3.client('ses', region_name='us-east-1')
message = {
'Subject': {'Data': 'AWS training account cost report'},
'Body': {'Html': {'Data': html}}
}
response = ses_client.send_email(
Source='Your Email id to send email',
Destination={'ToAddresses': [ 'your reciver email id']},
Message=message
)
if response and response.get('ResponseMetadata', {}).get('HTTPStatusCode') == 200:
success_message = 'Email has been sent successfully'
print(success_message)
return success_message
else:
error_message = 'Email sending failed. Check the SES response for details.'
print(error_message)
return error_message
if today.weekday() == 4:
print('week')
week = today - datetime.timedelta(days = 7)
str_week = str(week)
response_total = billing_client.get_cost_and_usage(
TimePeriod={
'Start': str_week,
'End': str_today },
Granularity='MONTHLY',
Metrics=[ 'UnblendedCost',]
)
print(response_total)
length=len(response_total["ResultsByTime"])
print(length)
if (length==2):
total_cost_1 = response_total["ResultsByTime"][0]['Total']['UnblendedCost']['Amount']
total_cost_2 = response_total["ResultsByTime"][1]['Total']['UnblendedCost']['Amount']
total_cost_1=float(total_cost_1)
total_cost_2=float(total_cost_2)
total_cost = total_cost_1+total_cost_2
total_cost=round(total_cost, 3)
total_cost = '$' + str(total_cost)
# print the total cost
print('Total cost for the week: ' + total_cost)
# get detailed billing for individual resources
response_detail = billing_client.get_cost_and_usage(
TimePeriod={
'Start': str_week,
'End': str_today
},
Granularity='MONTHLY',
Metrics=['UnblendedCost'],
GroupBy=[
{
'Type': 'DIMENSION',
'Key': 'SERVICE'
},
{
'Type': 'DIMENSION',
'Key': 'USAGE_TYPE'
}
]
)
resources = {'Service':[],'Usage Type':[],'Cost':[]}
resources_1 = {'Service':[],'Usage Type':[],'Cost':[]}
for result in response_detail['ResultsByTime'][0]['Groups']:
group_key = result['Keys']
service = group_key[0]
usage_type = group_key[1]
cost = result['Metrics']['UnblendedCost']['Amount']
cost=float(cost)
cost=round(cost, 3)
if cost > 0:
cost = '$' + str(cost)
resources['Service'].append(service)
resources['Usage Type'].append(usage_type)
resources['Cost'].append(cost)
for result in response_detail['ResultsByTime'][1]['Groups']:
group_key = result['Keys']
service = group_key[0]
usage_type = group_key[1]
cost = result['Metrics']['UnblendedCost']['Amount']
cost=float(cost)
cost=round(cost, 3)
if cost > 0:
cost = '$' + str(cost)
resources_1['Service'].append(service)
resources_1['Usage Type'].append(usage_type)
resources_1['Cost'].append(cost)
for key, value in resources_1.items():
if key in resources:
resources[key] += value
else:
resources[key] = value
else:
total_cost = response_total["ResultsByTime"][0]['Total']['UnblendedCost']['Amount']
total_cost=float(total_cost)
total_cost=round(total_cost, 3)
total_cost = '$' + str(total_cost)
# print the total cost
print('Total cost for the week: ' + total_cost)
# get detailed billing for individual resources
response_detail = billing_client.get_cost_and_usage(
TimePeriod={
'Start': str_week,
'End': str_today
},
Granularity='MONTHLY',
Metrics=['UnblendedCost'],
GroupBy=[
{
'Type': 'DIMENSION',
'Key': 'SERVICE'
},
{
'Type': 'DIMENSION',
'Key': 'USAGE_TYPE'
}
]
)
resources = {'Service':[],'Usage Type':[],'Cost':[]}
for result in response_detail['ResultsByTime'][0]['Groups']:
group_key = result['Keys']
service = group_key[0]
usage_type = group_key[1]
cost = result['Metrics']['UnblendedCost']['Amount']
cost=float(cost)
cost=round(cost, 3)
if cost > 0:
cost = '$' + str(cost)
resources['Service'].append(service)
resources['Usage Type'].append(usage_type)
resources['Cost'].append(cost)
print(type(resources))
df = pd.DataFrame(resources)
html_table = df.to_html(index=False)
print(resources)
message = 'Cost of AWS training account for the was'
html = """
<html>
<head>
<style>
body {{
font-family: Arial, sans-serif;
color: white;
background-color: black;
}}
h2 {{
color: white;
font-size: 25px;
text-align: center;
}}
h1 {{
color: #333333;
font-size: 40px;
text-align: center;
background-color: yellow;
}}
p {{
color: white;
font-size: 30px;
line-height: 1.5;
margin-bottom: 20px;
text-align: center;
}}
p1 {{
font-size: 10px;
text-align: center;
margin-left: auto;
margin-right: auto;
}}
</style>
</head>
<body>
<p> Training Account report for the week {} and {} </p>
<h2> {} </h2>
<h1> <strong> <em> {} </em></strong> </h1>
<p1>{}</p1>
</body>
</html>
""".format(str_week,str_today,message,total_cost,html_table)
ses_client = boto3.client('ses', region_name='us-east-1')
message = {
'Subject': {'Data': 'AWS training account cost report'},
'Body': {'Html': {'Data': html}}
}
response = ses_client.send_email(
Source='Your Email id to send email',
Destination={'ToAddresses': [ 'your reciver email id']},
Message=message
)
if response and response.get('ResponseMetadata', {}).get('HTTPStatusCode') == 200:
success_message = 'Email has been sent successfully'
print(success_message)
return success_message
else:
error_message = 'Email sending failed. Check the SES response for details.'
print(error_message)
return error_message
Now click on Deploy then Test. Create sample event and test it.
If you see "Email has been sent successfully" in Response then Bingo You must get email with yesterday cost report.
Step 5: Automation with EventBridge Rule
Create EventBridge Rule:
Set up an EventBridge rule to automate Lambda function execution.
Choose rule type as "Schedule" and set the trigger time.
Add Lambda as Target:
Add the Lambda function as a target for the EventBridge rule.
Then Create it. Bingo you will get last month report on specified date and time each month.
Conclusion:
Congratulations! You've successfully implemented a robust cost management solution on AWS. This guide provides detailed steps, ensuring a seamless setup of SES, Lambda function, and automation with EventBridge rules. Optimize your AWS cost management effortlessly with this comprehensive solution.
That was all for today's project. See you in another project.
