DevOps Buzz
Search…
Bash / Shell
Bitbucket
Distros
Elasticsearch
General
Guidelines / Standards
microk8s
Prometheus
RabbitMQ
VirtualBox
Cheat Sheet
Useful AWS tricks.

CLI

Autocomplete

1
complete -C '$(which aws_completer)' aws
Copied!

References

ECR

Push Docker image.

Setup IAM policy

Make sure your IAM user has the following policy attached to it.
1
AmazonEC2ContainerRegistryFullAccess
Copied!

Create a repository

Go to your AWS console, Services, Elastic Container Service.
Click on Create Repository.
Input your repository name and click on Next Step. AWS console will show pretty much the same instructions bellow.

AWS ECR login

Export your AWS credentials.
To retieve AWS ECR login command, run:
1
aws ecr get-login --no-include-email --region $AWS_DEFAULT_REGION
Copied!
The return will be something like this:
1
docker login -u AWS -p eyWwBXlsg2FkIjoiL2hreVdiRWloWEM0NHFGcStaQ0hmYVZQOGJCVFFwODZPYXRrekZyaUZXY21pRVZNL09qMHMzRFM2S2Y5MkFleGJhYUxjeW1GR2srTFc5VUYxejBVWFFpakVuclhqM2JJeWN6MDh2d3pGZVE1YXA0NVYrSnArRDZCNkdpQWhuK3AycUhmdEpKTU1yM2srNFZ5OWdheC85UHN5WmhVSDJZUklwRC9tYTQ2MWdLNUVjNnpycWovMytOY2V2RTNkamxNTWg1MXBCMkpMVG9TM3prWUhacDN2b1IzazBDM25qV2thVlNDYy9NbnFxTTJjaHdvQWEwbG1pUnU0ZGVUN1BRWXhZUUVUb1VsNjZkR1RZcGdtZ2RBbWh0TXZTbjcwWVN1c2djVEhtV3dLVGF3VjlRcjh1d21Ybm0vbExqL2VmZm04NllXeUxBaS9zb3g5akhIMHZsT2tydlZ5bEtUT3VySFNxQWdFamRCSndQM1F4U1gzd1lBc0x6Uzk0andiTW41VWwxTXYzNzZDVkxZc09lRVBjM1UxM0tuMFhSMGt3Y0V1L282SWlyc1BUSGh5Tnc1THF0S0swLy8vWm5SSWlTWFhhaWEvbU50c3J6OWhYRWFqV1VRVjQ4M09sbGpVT0JnUzJhdm9MSkkyU2tVSjRiMzBrdnVEOGNJVjVTajk3QzJRNzk4Q2ppUWVzV2JuV2RMN29Zd1JPbHc2dVViTlhQb244V1QwWDdPVjQyandmSXFuL1A3ODJad2p3a2QrVFVBNTRYc1UrQ1dPcmF6VG9oMDZ3OWczVmVZZnJJc0RaOHNIYnVZYkoxaDZwV1VsU2RZYURDTXJwRE9SUEN5YUpBWFFsVlZ1TWJJR2ZacXQ2bGRORXdFR0NVd1QrUWZpSzNuR0RCNkZVREx3Q0FEcGJoVEdjRllpWDIyVENVd0JHUkc5aGFSaU00NXV6V3pVLytJNk9TOFZPdmlSdTA5WHQxTms2cmNEeHJ4OE5DR0pNTGJkdGFLSVI3MW9aY1BYNS9JejJDdDZzYUZvekczVTQrYjZFV1ErbnRUeFZGakxRWmxwOTN1TUpVc3VyTDFQK3Y0MnBsSnNENXhTeFhHdy80aFhjVUlzOElQQkRmRjZEYWJxNC8vZCtMOVRKS2ZJakpwRXdrc1A3UVp3UVJWdURPM1lGV2NiakhvMmxUMXkwQ3RIUWNXZkRVeS9uMitPZHZIWmd1b0NSaU45elRoa0haRE9lcmtFU3FIR3hwK2x2eXpSMnR6NSs0VXhjQTJlR1FJaTFuVnB6MWFnbzY1cTM1WW1JREFIOUMvUDN2WmVZbVRpQTIxRW1tVnZaMURCK1hVbVlwUURkbGpOMW82SUZRbktzWGRlU0FFMTFsTUNxVlFnTXJUS25BbWhDTzIvMStBLzNKVmNEbC9pY1FCWGlvUFhXVmRlNU1WK2VnN3ViSWJyVGw0clVRdkt0VlFXdWVSdmVYWUhMeTlTa2lqWXhZSkZHTU53cnFMeERpWFhxOGIrQjZLVlFrcnVnPT0iLCJkYXRha2V5IjoiQVFFQkFIaEx5ano3UEpFUG1ZYnFINkE2Zi93V056eXo1QkthMzdCbkJQc0F5dUYxS1FBQUFINHdmQVlKS29aSWh2Y05BUWNHb0c4d2JRSUJBREJvQmdrcWhraUc5dzBCQndFd0hnWUpZSVpJQVdVREJBRXVNQkVFRFA5SWFOV0FBTFgvd3pBYUlnSUJFSUE3VlFiY3NkU3Z4UjMzZ1I0WDRPbldQTnBScFRhTythdFRWajFPQXErNFpjOWM0eGhqeTJ2UmtjaFJIcmdxUERycmI4OFZQZmNITEgrUkxCZz0iLCJ2ZXJzaW9uIjoiMiIsInR5cGUiOiJEQVRBX0tFWSIsImV4cGlyYXRpb24iOjE1Mjg0MzU3MTB9 https://189013708713.dkr.ecr.ap-southeast-2.amazonaws.com
Copied!
Copy and run your login command.

Push your Docker image to AWS ECR

Tag your Docker image (replace with your info):
1
docker tag \
2
YOUR-IMAGE-NAME:latest \
3
189013708713.dkr.ecr.ap-southeast-2.amazonaws.com/YOUR-IMAGE-NAME:latest
Copied!
Then push it to AWS ECR.
1
docker push \
2
189013708713.dkr.ecr.ap-southeast-2.amazonaws.com/YOUR-IMAGE-NAME:latest
Copied!

Lambda

Allow lambda function to access RDS

VPC and Security Groups

Create a Security Group that will be used for your lambda function.
Go to your lambda function setup on the AWS console.
Put the Lambda function in the same RDS VPC.
Add the Lambda function in all subnets.
Add the Security Group you just created.
In your RDS security group, allow your lambda function security group.

VPC endpoint

Go to Services -> VPC -> Endpoints -> Create Endpoint.
Select AWS Services as the Service category, and then, in the Service Name list, select the Secrets Manager endpoint service named com.amazonaws.us-west-2.secretsmanager
Could not load image
Specify the VPC you want to create the endpoint in. For this post, I chose the VPC named vpc-5ad42b3c where my RDS instance and application are running.
To create a VPC endpoint, you need to specify the private IP address range in which the endpoint will be accessible. To do this, select the subnet for each Availability Zone (AZ). This restricts the VPC endpoint to the private IP address range specific to each AZ and also creates an AZ-specific VPC endpoint. Specifying more than one subnet-AZ combination helps improve fault tolerance and make the endpoint accessible from a different AZ in case of an AZ failure. Here, I specify subnet IDs for availability zones us-west-2a, us-west-2b, and us-west-2c:
Select the Enable Private DNS Name checkbox for the VPC endpoint. Private DNS resolves the standard Secrets Manager DNS hostname https://secretsmanager..amazonaws.com. to the private IP addresses associated with the VPC endpoint specific DNS hostname. As a result, you can access the Secrets Manager VPC Endpoint via the AWS Command Line Interface (AWS CLI) or AWS SDKs without making any code or configuration changes to update the Secrets Manager endpoint URL.
Associate a security group with this endpoint. The security group enables you to control the traffic to the endpoint from resources in your VPC. For this post, I chose to associate the security group named sg-07e4197d that I created earlier. This security group has been set up to allow all instances running within VPC vpc-5ad42b3c to access the Secrets Manager VPC endpoint. Select Create endpoint to finish creating the endpoint.
To view the details of the endpoint you created, select the link on the console.
The Details tab shows all the DNS hostnames generated while creating the Amazon VPC endpoint that can be used to connect to Secrets Manager. I can now use the standard endpoint secretsmanager.us-west-2.amazonaws.com or one of the VPC-specific endpoints to connect to Secrets Manager within vpc-5ad42b3c where my RDS instance and application also resides.
Now that I have created the VPC endpoint, all traffic between my application running on an EC2 instance hosted within VPC named vpc-5ad42b3c and Secrets Manager will be within VPC. This connection will use the VPC endpoint and I can use it to retrieve my RDS database secret stored in Secrets Manager. I can retrieve the secret via the AWS SDK or CLI. As an example, I can use the CLI command shown below to retrieve the current version of my RDS database secret:
1
aws secretsmanager \
2
get-secret-value \
3
–secret-id \
4
MyDatabaseSecret \
5
–version-stage AWSCURRENT
Copied!
Since my AWS CLI is configured for us-west-2 region, it uses the standard Secrets Manager endpoint URL https://secretsmanager.us-west-2.amazonaws.com. This standard endpoint automatically routes to the VPC endpoint since I enabled support for Private DNS hostname while creating the VPC endpoint. The above command will result in the following output:
1
{
2
"ARN": "arn:aws:secretsmanager:us-west-2:123456789012:secret:MyDatabaseSecret-a1b2c3",
3
"Name": "MyDatabaseSecret",
4
"VersionId": "EXAMPLE1-90ab-cdef-fedc-ba987EXAMPLE",
5
"SecretString": "{\n \"username\":\"david\",\n \"password\":\"BnQw&XDWgaEeT9XGTT29\"\n}\n",
6
"VersionStages": [
7
"AWSCURRENT"
8
],
9
"CreatedDate": 1523477145.713
10
}
Copied!

Test

Go to AWS Secrets Manager -> Secrets -> YOUR SECRET
Get the examples in "Sample code" and add to your lambda function code.

References

S3

Create bucket

1
aws s3 mb s3://BUCKET-NAME
Copied!

List bucket content

1
aws s3 ls s3://YOUR-BUCKET-NAME
Copied!

General S3 management

Get bucket content size

1
aws s3 ls \
2
--summarize \
3
--human-readable \
4
--recursive \
5
s3://BUCKET-NAME
Copied!

Copy from bucket

1
aws s3 cp \
2
--recursive \
3
s3://BUCKET-NAME \
4
DESTINATION-FOLDER
Copied!

Copy to bucket

1
aws s3 cp \
2
--recursive \
3
FILE-OR-FOLDER \
4
s3://BUCKET-NAME
Copied!

Synchronize content from bucket

1
aws s3 sync \
2
LOCAL-FOLDER \
3
s3://BUCKET-NAME
Copied!

Volumes

Attach EBS volume to EC2 instance

Head over to EC2 –> Volumes and create a new volume of your preferred size and type.
Select the created volume, right click and select the "attach volume" option.
Select the instance from the instance text box as shown below.
Now, login to your ec2 instance and list the available disks using the following command.
1
lsblk
Copied!
Example output:
1
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
2
loop0 7:0 0 91M 1 loop /snap/core/6405
3
loop1 7:1 0 12.7M 1 loop /snap/amazon-ssm-agent/495
4
loop2 7:2 0 17.9M 1 loop /snap/amazon-ssm-agent/1068
5
loop3 7:3 0 18M 1 loop /snap/amazon-ssm-agent/930
6
loop4 7:4 0 91.1M 1 loop /snap/core/6259
7
loop5 7:5 0 91M 1 loop /snap/core/6350
8
xvda 202:0 0 300G 0 disk
9
└─xvda1 202:1 0 300G 0 part /
10
xvdf 202:80 0 1000G 0 disk
Copied!
In this case, the device name is xvdf
Make sure the volume has no data:
1
file -s /dev/xvdf
Copied!
Example output:
1
/dev/xvdf: data
Copied!
Format the disk:
1
mkfs.ext4 /dev/xvdf
Copied!
Label the disk:
1
e2label /dev/xvdf "storage-001"
Copied!
Mount the disk:
1
mkdir -p /storage/storage-001
2
mount /dev/xvdf /storage/storage-001
Copied!
Configure fstab:
1
cp /etc/fstab /etc/fstab-BKP
2
nano /etc/fstab
Copied!
Add:
1
LABEL=storage-001 /storage/storage-001 ext4 defaults,discard 0 0
Copied!
Test fstab:
1
umount /storage/storage-001
2
mount -a
3
df -h
Copied!

Resize volume from snapshot

Create a snapshot of the volume you want to increase size.
Create a new volume from the snapshot.
Attach the new volume.
SSH to the instance an run:
1
e2fsck -f /dev/xvdh
2
resize2fs /dev/xvdh
Copied!
Umount current volume:
1
umount /data
Copied!
Edit fstab, comment out the old volume and add the new one:
1
#/dev/xvdg /data ext4 defaults 0 0
2
/dev/xvdh /data ext4 defaults 0 0
Copied!
Mount it:
1
mount -a
Copied!