AWS IAM and Cost Explorer CLI Setup

While using Amazon EC2 for my class on System Administration, students often leave resources running unchecked, leading to an unexpectedly high bill. In order to help them keep track of their costs, I put together a simple shell function to display the current usage.

While the command itself (shown at the end of this post) is trivial by itself, creating the required AWS resources to allow for access via the command-line -- our ~~preferred~~ mandated AWS access method -- is not completely obvious for novice or even only occasional AWS users, so I figured I'd document this here, both for my students as well as for myself.

Create a dedicated IAM user

First, we create a new user to access IAM via the command-line. This is one of the few instances where we access AWS via the browser UI: we don't have access to manipulate IAM via the command-line yet, as that is exactly what we're setting up here.

So:

log in to the AWS Console, and select "IAM" as the service
select "Users" -> "Add users"
add a user name, e.g., "iam-admin"
on the next screen, select "Attach policies directly"
select "create policy", which will open in a new browser tab
"Choose a service" -> IAM
select "All IAM actions (iam:*)"
if there are warnings, select the "Resources" and select "Any in this account" on the right-hand side for each
"Next: Tags", then "Next: Review"
give the policy a name, e.g., "iam-admin"
"create policy"
go back to the previous tab where we are creating the new user
under "Permissions Policy", click the refresh loop arrows 🔄
search for "iam-admin" and select the new policy
"next", then finally "create user"

Now we have a new user with full IAM privileges, but we still need to create our CLI credentials and set everything up so we can finally abandon the browser. To do that:

With that browser tab open, edit your ~/.aws/credentials file to add the access credentials and configure a separate AWS profile for your CLI, and finally verify your IAM access:

$ cat >> ~/.aws/credentials <<EOF

[iam-admin]
aws_access_key_id = the-access-key-you-copied-from-the-browser
aws_secret_access_key= the-secret-access-key-you-copied-from-the-browser
EOF
$ cat >> ~/.aws/config <<EOF

[profile iam-admin]
region = us-east-1
output = json
EOF
$ AWS_PROFILE=iam-admin aws iam get-user | jq -r '.User.UserName'
iam-admin
$

Grant Cost Explorer Permissions

Now that we can make changes in IAM from the command-line, we can extend the AWS Cost Explorer privileges to the user we use for normal operations. For this, we assume that you have already created a user in IAM for use with EC2 with the right privileges. (If you haven't, you can now use aws iam create-user etc. to do so!)

First, we'll need to create a new policy to allow read-only access to "Cost Explorer":

$ export AWSID=$(aws sts get-caller-identity | jq -r .Account)
$ cat >policy <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ce:DescribeCostCategoryDefinition",
                "ce:ListTagsForResource",
                "ce:GetAnomalySubscriptions",
                "ce:GetAnomalies",
                "ce:GetAnomalyMonitors"
            ],
            "Resource": [
                "arn:aws:ce::${AWSID}:anomalysubscription/*",
                "arn:aws:ce::${AWSID}:costcategory/*",
                "arn:aws:ce::${AWSID}:anomalymonitor/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ce:GetRightsizingRecommendation",
                "ce:GetCostAndUsage",
                "ce:GetSavingsPlansUtilization",
                "ce:GetReservationPurchaseRecommendation",
                "ce:ListCostCategoryDefinitions",
                "ce:GetCostForecast",
                "ce:ListSavingsPlansPurchaseRecommendationGeneration",
                "ce:GetPreferences",
                "ce:GetReservationUtilization",
                "ce:GetCostCategories",
                "ce:GetSavingsPlansPurchaseRecommendation",
                "ce:GetDimensionValues",
                "ce:GetSavingsPlansUtilizationDetails",
                "ce:GetCostAndUsageWithResources",
                "ce:ListCostAllocationTags",
                "ce:DescribeReport",
                "ce:GetReservationCoverage",
                "ce:GetSavingsPlansCoverage",
                "ce:DescribeNotificationSubscription",
                "ce:GetTags",
                "ce:GetUsageForecast"
            ],
            "Resource": "*"
        }
    ]
}
EOF
$

This policy may be more than what we need immediately, but it covers all read operations we may eventually wish to use. Tailor it to your specific needs, if you like.

Next, we create the policy "cost-explorer", a new IAM group (also named "cost-explorer"), attach the new policy to the new group, and finally add our default CLI user (jschauma in my case) to this new group:

$ AWS_PROFILE=iam-admin aws iam create-policy --policy-name cost-explorer      \
        --policy-document file://policy
$ AWS_PROFILE=iam-admin aws iam create-group --group-name cost-explorer
$ export AWSID=$(aws sts get-caller-identity | jq -r .Account)
$ AWS_PROFILE=iam-admin aws iam attach-group-policy --group-name cost-explorer \
        --policy-arn arn:aws:iam::${AWSID}:policy/cost-explorer
$ AWS_PROFILE=iam-admin aws iam add-user-to-group --group-name cost-explorer   \
        --user-name jschauma
$

If all that went through without any problems, then we should now be in business and be able to use our default user to query Cost Explorer. For that, we need to specify a start and end date, so let's just pick the last 30 days as the time window and round the total to two digits:

$ start=$(date -r $(( $(date +%s) - (86400 * 30) )) +%Y-%m-%d)
$ end=$(date +%Y-%m-%d)
$ aws ce get-cost-and-usage --output json                   \
        --time-period Start=${start},End=${end}             \
        --granularity MONTHLY                               \
        --metrics UnblendedCost                             \
        --query 'ResultsByTime[*].Total.[UnblendedCost]' |  \
        jq '.[][0].Amount | tonumber*100 | round/100'

Listing EC2 resources

Now in addition to knowing how much you owe Amazon, it's also useful to keep track of your currently used resources. For my specific purposes here, students tend to not have more than just a handful of instances of volumes in use, so we can list them all easily by describing all AMIs, snapshots, instances, and volumes.

Putting this all together, I now added a function awsCurrentBill() as well as one called awsListResources() to my various AWS aliases and have it run from my shell's startup files at login time on our class's shared shell server:

$ ssh shell
NetBSD 9.3 (GENERIC) #0: Tue Aug  9 17:50:54 EDT 2022

Welcome to NetBSD!

You have the following AMIs:
ami-0669dd7b1ff900fcc snap-0c37190f3fa002bc7 "OmniOS illumos distribution"
ami-08b87fed21cce91cb snap-0b149542eaa295feb "NetBSD/evbarm 9.1"
ami-05ffda7ac6da57de1 snap-032e8bb49d60ca82a "NetBSD/amd64 -current (10.99.2) built on 2023-01-23"

You have the following snapshots:
snap-0b149542eaa295feb 2021-01-22T03:42:18.235000+00:00
snap-0c37190f3fa002bc7 2023-01-27T00:34:13.769000+00:00
snap-032e8bb49d60ca82a 2023-01-27T02:43:19.716000+00:00

You have the following instances:
i-0608051b72e7ef9db (stopped) 
i-0782730ccc4715104 (running) ec2-54-162-249-177.compute-1.amazonaws.com
i-0c7f26b7bf6a8340b (terminated) 

You have the following volumes:
vol-03eda8dcb95fe19be 2023-01-30T20:38:24.313000+00:00 i-0608051b72e7ef9db
vol-0372aefbe6efe23a7 2023-02-01T14:54:36.002000+00:00 i-0782730ccc4715104
vol-0a7d5e315be306df4 2023-02-01T14:54:23.467000+00:00 i-0782730ccc4715104
$ awsCurrentBill
AWS Billing Period 2023-01-01 - 2023-01-31: $4.62
$

Note: AWS Cost Explorer charges $0.01 per request, so adding awsCurrentBill to your ~/.profile might be too much. Either way, hopefully this will help students avoid running up a large AWS bill by accident.

Signs of Triviality