Cheat Sheet

Useful AWS tricks.

CLI

Autocomplete

complete -C '$(which aws_completer)' aws

References

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-completion.html

ECR

Push Docker image.

Setup IAM policy

Make sure your IAM user has the following policy attached to it.

AmazonEC2ContainerRegistryFullAccess

Create a repository

Go to your AWS console, Services, Elastic Container Service.

Click on Create Repository.

Input your repository name and click on Next Step. AWS console will show pretty much the same instructions bellow.

AWS ECR login

Export your AWS credentials.

To retieve AWS ECR login command, run:

aws ecr get-login --no-include-email --region $AWS_DEFAULT_REGION

The return will be something like this:

docker login -u AWS -p eyWwBXlsg2FkIjoiL2hreVdiRWloWEM0NHFGcStaQ0hmYVZQOGJCVFFwODZPYXRrekZyaUZXY21pRVZNL09qMHMzRFM2S2Y5MkFleGJhYUxjeW1GR2srTFc5VUYxejBVWFFpakVuclhqM2JJeWN6MDh2d3pGZVE1YXA0NVYrSnArRDZCNkdpQWhuK3AycUhmdEpKTU1yM2srNFZ5OWdheC85UHN5WmhVSDJZUklwRC9tYTQ2MWdLNUVjNnpycWovMytOY2V2RTNkamxNTWg1MXBCMkpMVG9TM3prWUhacDN2b1IzazBDM25qV2thVlNDYy9NbnFxTTJjaHdvQWEwbG1pUnU0ZGVUN1BRWXhZUUVUb1VsNjZkR1RZcGdtZ2RBbWh0TXZTbjcwWVN1c2djVEhtV3dLVGF3VjlRcjh1d21Ybm0vbExqL2VmZm04NllXeUxBaS9zb3g5akhIMHZsT2tydlZ5bEtUT3VySFNxQWdFamRCSndQM1F4U1gzd1lBc0x6Uzk0andiTW41VWwxTXYzNzZDVkxZc09lRVBjM1UxM0tuMFhSMGt3Y0V1L282SWlyc1BUSGh5Tnc1THF0S0swLy8vWm5SSWlTWFhhaWEvbU50c3J6OWhYRWFqV1VRVjQ4M09sbGpVT0JnUzJhdm9MSkkyU2tVSjRiMzBrdnVEOGNJVjVTajk3QzJRNzk4Q2ppUWVzV2JuV2RMN29Zd1JPbHc2dVViTlhQb244V1QwWDdPVjQyandmSXFuL1A3ODJad2p3a2QrVFVBNTRYc1UrQ1dPcmF6VG9oMDZ3OWczVmVZZnJJc0RaOHNIYnVZYkoxaDZwV1VsU2RZYURDTXJwRE9SUEN5YUpBWFFsVlZ1TWJJR2ZacXQ2bGRORXdFR0NVd1QrUWZpSzNuR0RCNkZVREx3Q0FEcGJoVEdjRllpWDIyVENVd0JHUkc5aGFSaU00NXV6V3pVLytJNk9TOFZPdmlSdTA5WHQxTms2cmNEeHJ4OE5DR0pNTGJkdGFLSVI3MW9aY1BYNS9JejJDdDZzYUZvekczVTQrYjZFV1ErbnRUeFZGakxRWmxwOTN1TUpVc3VyTDFQK3Y0MnBsSnNENXhTeFhHdy80aFhjVUlzOElQQkRmRjZEYWJxNC8vZCtMOVRKS2ZJakpwRXdrc1A3UVp3UVJWdURPM1lGV2NiakhvMmxUMXkwQ3RIUWNXZkRVeS9uMitPZHZIWmd1b0NSaU45elRoa0haRE9lcmtFU3FIR3hwK2x2eXpSMnR6NSs0VXhjQTJlR1FJaTFuVnB6MWFnbzY1cTM1WW1JREFIOUMvUDN2WmVZbVRpQTIxRW1tVnZaMURCK1hVbVlwUURkbGpOMW82SUZRbktzWGRlU0FFMTFsTUNxVlFnTXJUS25BbWhDTzIvMStBLzNKVmNEbC9pY1FCWGlvUFhXVmRlNU1WK2VnN3ViSWJyVGw0clVRdkt0VlFXdWVSdmVYWUhMeTlTa2lqWXhZSkZHTU53cnFMeERpWFhxOGIrQjZLVlFrcnVnPT0iLCJkYXRha2V5IjoiQVFFQkFIaEx5ano3UEpFUG1ZYnFINkE2Zi93V056eXo1QkthMzdCbkJQc0F5dUYxS1FBQUFINHdmQVlKS29aSWh2Y05BUWNHb0c4d2JRSUJBREJvQmdrcWhraUc5dzBCQndFd0hnWUpZSVpJQVdVREJBRXVNQkVFRFA5SWFOV0FBTFgvd3pBYUlnSUJFSUE3VlFiY3NkU3Z4UjMzZ1I0WDRPbldQTnBScFRhTythdFRWajFPQXErNFpjOWM0eGhqeTJ2UmtjaFJIcmdxUERycmI4OFZQZmNITEgrUkxCZz0iLCJ2ZXJzaW9uIjoiMiIsInR5cGUiOiJEQVRBX0tFWSIsImV4cGlyYXRpb24iOjE1Mjg0MzU3MTB9 https://189013708713.dkr.ecr.ap-southeast-2.amazonaws.com

Copy and run your login command.

Push your Docker image to AWS ECR

Tag your Docker image (replace with your info):

docker tag \
  YOUR-IMAGE-NAME:latest \
  189013708713.dkr.ecr.ap-southeast-2.amazonaws.com/YOUR-IMAGE-NAME:latest

Then push it to AWS ECR.

docker push \
  189013708713.dkr.ecr.ap-southeast-2.amazonaws.com/YOUR-IMAGE-NAME:latest

Lambda

Allow lambda function to access RDS

VPC and Security Groups

Create a Security Group that will be used for your lambda function.

Go to your lambda function setup on the AWS console.

Put the Lambda function in the same RDS VPC.

Add the Lambda function in all subnets.

Add the Security Group you just created.

In your RDS security group, allow your lambda function security group.

VPC endpoint

Go to Services -> VPC -> Endpoints -> Create Endpoint.

Select AWS Services as the Service category, and then, in the Service Name list, select the Secrets Manager endpoint service named com.amazonaws.us-west-2.secretsmanager

Specify the VPC you want to create the endpoint in. For this post, I chose the VPC named vpc-5ad42b3c where my RDS instance and application are running.

To create a VPC endpoint, you need to specify the private IP address range in which the endpoint will be accessible. To do this, select the subnet for each Availability Zone (AZ). This restricts the VPC endpoint to the private IP address range specific to each AZ and also creates an AZ-specific VPC endpoint. Specifying more than one subnet-AZ combination helps improve fault tolerance and make the endpoint accessible from a different AZ in case of an AZ failure. Here, I specify subnet IDs for availability zones us-west-2a, us-west-2b, and us-west-2c:

Select the Enable Private DNS Name checkbox for the VPC endpoint. Private DNS resolves the standard Secrets Manager DNS hostname https://secretsmanager..amazonaws.com. to the private IP addresses associated with the VPC endpoint specific DNS hostname. As a result, you can access the Secrets Manager VPC Endpoint via the AWS Command Line Interface (AWS CLI) or AWS SDKs without making any code or configuration changes to update the Secrets Manager endpoint URL.

Associate a security group with this endpoint. The security group enables you to control the traffic to the endpoint from resources in your VPC. For this post, I chose to associate the security group named sg-07e4197d that I created earlier. This security group has been set up to allow all instances running within VPC vpc-5ad42b3c to access the Secrets Manager VPC endpoint. Select Create endpoint to finish creating the endpoint.

To view the details of the endpoint you created, select the link on the console.

The Details tab shows all the DNS hostnames generated while creating the Amazon VPC endpoint that can be used to connect to Secrets Manager. I can now use the standard endpoint secretsmanager.us-west-2.amazonaws.com or one of the VPC-specific endpoints to connect to Secrets Manager within vpc-5ad42b3c where my RDS instance and application also resides.

Now that I have created the VPC endpoint, all traffic between my application running on an EC2 instance hosted within VPC named vpc-5ad42b3c and Secrets Manager will be within VPC. This connection will use the VPC endpoint and I can use it to retrieve my RDS database secret stored in Secrets Manager. I can retrieve the secret via the AWS SDK or CLI. As an example, I can use the CLI command shown below to retrieve the current version of my RDS database secret:

aws secretsmanager \
  get-secret-value \
  –secret-id \
  MyDatabaseSecret \
  –version-stage AWSCURRENT

Since my AWS CLI is configured for us-west-2 region, it uses the standard Secrets Manager endpoint URL https://secretsmanager.us-west-2.amazonaws.com. This standard endpoint automatically routes to the VPC endpoint since I enabled support for Private DNS hostname while creating the VPC endpoint. The above command will result in the following output:

{
  "ARN": "arn:aws:secretsmanager:us-west-2:123456789012:secret:MyDatabaseSecret-a1b2c3",
  "Name": "MyDatabaseSecret",
  "VersionId": "EXAMPLE1-90ab-cdef-fedc-ba987EXAMPLE",
  "SecretString": "{\n  \"username\":\"david\",\n  \"password\":\"BnQw&XDWgaEeT9XGTT29\"\n}\n",
  "VersionStages": [
    "AWSCURRENT"
  ],
  "CreatedDate": 1523477145.713
} 

Test

Go to AWS Secrets Manager -> Secrets -> YOUR SECRET

Get the examples in "Sample code" and add to your lambda function code.

References

https://docs.aws.amazon.com/lambda/latest/dg/vpc-rds.html

https://aws.amazon.com/blogs/security/how-to-connect-to-aws-secrets-manager-service-within-a-virtual-private-cloud/

S3

Create bucket

aws s3 mb s3://BUCKET-NAME

List bucket content

aws s3 ls s3://YOUR-BUCKET-NAME

General S3 management

s4cmd: https://github.com/bloomreach/s4cmd

Get bucket content size

aws s3 ls \
--summarize \
--human-readable \
--recursive \
s3://BUCKET-NAME

Copy from bucket

aws s3 cp \
--recursive \
s3://BUCKET-NAME \
DESTINATION-FOLDER

Copy to bucket

aws s3 cp \
--recursive \
FILE-OR-FOLDER \
s3://BUCKET-NAME

Synchronize content from bucket

aws s3 sync \
LOCAL-FOLDER \
s3://BUCKET-NAME

Volumes

Attach EBS volume to EC2 instance

Head over to EC2 –> Volumes and create a new volume of your preferred size and type.

Select the created volume, right click and select the "attach volume" option.

Select the instance from the instance text box as shown below.

Now, login to your ec2 instance and list the available disks using the following command.

lsblk

Example output:

NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0     7:0    0   91M  1 loop /snap/core/6405
loop1     7:1    0 12.7M  1 loop /snap/amazon-ssm-agent/495
loop2     7:2    0 17.9M  1 loop /snap/amazon-ssm-agent/1068
loop3     7:3    0   18M  1 loop /snap/amazon-ssm-agent/930
loop4     7:4    0 91.1M  1 loop /snap/core/6259
loop5     7:5    0   91M  1 loop /snap/core/6350
xvda    202:0    0  300G  0 disk 
└─xvda1 202:1    0  300G  0 part /
xvdf    202:80   0 1000G  0 disk

In this case, the device name is xvdf

Make sure the volume has no data:

file -s /dev/xvdf 

Example output:

/dev/xvdf: data

Format the disk:

mkfs.ext4 /dev/xvdf

Label the disk:

e2label /dev/xvdf "storage-001"

Mount the disk:

mkdir -p /storage/storage-001
mount /dev/xvdf /storage/storage-001

Configure fstab:

cp /etc/fstab /etc/fstab-BKP
nano /etc/fstab

Add:

LABEL=storage-001   /storage/storage-001        ext4   defaults,discard        0 0

Test fstab:

umount /storage/storage-001
mount -a
df -h

Resize volume from snapshot

Create a snapshot of the volume you want to increase size.

Create a new volume from the snapshot.

Attach the new volume.

SSH to the instance an run:

e2fsck -f /dev/xvdh
resize2fs /dev/xvdh

Umount current volume:

umount /data

Edit fstab, comment out the old volume and add the new one:

#/dev/xvdg /data ext4 defaults 0 0
/dev/xvdh /data ext4 defaults 0 0

Mount it:

mount -a