[AWS] Practical Operation-Securing Amazon S3 VPC Endpoint Communication

Article directory

• • Experiment overview
  • Target
  • lab environment
  • Task 1: Explore and launch the experimental environment
  • • Task 1.1: Explore Amazon VPC resources
    • Task 1.2: Explore Amazon EC2 resources
    • Task 1.3: Create an Amazon VPC endpoint
    • Task 1.4: Connect a private EC2 instance
    • Task 1.5: Explore Amazon S3 resources
  • Task 2: Interact with Amazon S3 through an Amazon EC2 instance
  • Task 3: Create a VPC endpoint for Amazon S3
  • Task 4: Interact with Amazon S3 through a private instance
  • Add VPC endpoint policy
  • solution
  • • Console steps
    • AWS CLI steps
  • Summarize

Experiment Overview

Data security is of the utmost importance. Amazon Web Services (AWS) provides a variety of solutions and best practices to help keep your data secure. Understanding how to make the best decisions to keep your data safe can be a challenging task. Running applications on a private subnet with no routes to the Internet limits the attack surface to internal traffic only, keeping data secure. This policy is a good security measure. However, problems can arise when applications must access data from services such as Amazon Simple Storage Service (Amazon S3) .

To solve this problem, AWS provides the Amazon Virtual Private Cloud (Amazon VPC) endpoint. Using VPC endpoints, you can privately connect your VPC to supported AWS services. **This private connection does not require an Internet gateway, NAT gateway, VPN connection, or AWS Direct Connect connection. **When communicating through a VPC endpoint, you do not need to use resources in the VPC to obtain a public IP address. This way, traffic through the VPC endpoint remains within the Amazon network.

In this lab, you create a VPC endpoint. These endpoints are then used to access Amazon S3 from an Amazon Elastic Compute Cloud (Amazon EC2) instance located in the private subnet. To further increase data security, you can create a VPC endpoint policy to restrict endpoint usage to specific resources.

Goals

After completing this lab, you will be able to:

• Understand private and public subnets and why they can or cannot communicate with Amazon S3.
• Configure VPC endpoints using the AWS Management Console and the AWS Command Line Interface (AWS CLI).
• Interact with Amazon S3 through a VPC endpoint in a private subnet.
• Create a VPC endpoint policy to restrict access to resources

Experimental environment

In this lab, a VPC has been created with public and private subnets. Private subnet route tables do not include NAT gateways or Internet gateways. Therefore, resources launched in a private subnet cannot communicate with the public internet or any AWS services that transmit data over the public internet.

This lab demonstrates how VPC endpoints work. To do this, an EC2 instance has been launched in the Public subnet and the same EC2 instance has been launched in the private subnet. VPC endpoints allow traffic from private EC2 instances to access AWS services that require public routing. To verify this result, this lab has created an Amazon S3 bucket containing a demo file.

The image below shows all the resources provisioned for this experiment:

Task 1: Explore and launch the experimental environment

In this task, you will view the pre-configured experiment resources. These resources include VPCs, subnets, Amazon S3 buckets, and Amazon EC2 instances. You can also create a VPC endpoint to allow AWS Systems Manager Session Manager to access the instance.

The following figure shows all the resources provisioned for this lab task and how they are connected:

Task 1.1: Explore Amazon VPC resources

• View VPC–labVPC

• View the subnets created in labVPC

• The public subnet PublicSubnetA is configured with an Internet gateway and can communicate externally.

• The private subnet PrivateSubnetA is not configured with an Internet gateway and cannot communicate externally.

• View routing table PublicRouteTable subnet association status

• View routing table PrivateRouteTable subnet association status

Task 1.2: Explore Amazon EC2 resources

Through the console, navigate to the EC2 control panel to view the cloud instance.

• View cloud instances

• Check the HTTPS-SSM-ENDPOINT security group used by the cloud host.

• This instance is a cloud host instance in the public network. You can use Session Manager to connect directly to the instance.

• You can connect to the cloud host instance through Session Manager.

ls

• Next, use the same method to log in to the cloud host instance. Found that I couldn’t log in.

An error message appears: "We weren't able to connect to your instance.". At this point, to connect to the instance using Session Manager, you can create an interface endpoint.

Systems Manager can be configured to use interface virtual private cloud (VPC) endpoints to further improve the security posture of managed nodes. AWS PrivateLink restricts all network traffic between managed node , Systems Manager, and Amazon EC2 to the Amazon network. This restriction eliminates the need for any outbound internet access on your private EC2 instance.

Task 1.3: Create an Amazon VPC endpoint

• Create a Session Manager endpoint to be able to connect to the private EC2 instance without using an internet gateway or NAT gateway.

• Endpoint configuration

1. Enter SSM Endpoint in the Name tag – optional box.
2. In the Service Name filter box, enter ssm and press Enter.
3. Select com.amazonaws.REGION.ssm, where REGION is the region from which you launched the experiment.
4. For VPC, select labVPC from the drop-down menu.

• It can be accessed and used after the terminal node status changes to Available status.

Task 1.4: Connect a private EC2 instance

• Select Connect to the PrivateCommandHost cloud instance.

• Select the *Connect* button. A new browser tab will open. You now have an instance endpoint connected to the instance in the private subnet.

• Connected successfully.

Task 1.5: Explore Amazon S3 resources

• View the bucket.

Task 2: Interacting with Amazon S3 through an Amazon EC2 instance

Connect to the public and private instances and try to access the files in the Amazon S3 bucket from each instance without using the Amazon S3 VPC endpoint.

As shown in the following figure, the S3 bucket is accessible from the public instance instead of the private instance:

• Command: To access your home directory, run the following command:

cd ~

Command: To help differentiate between commands and output in the AWS CLI, run the following command. This command will add a blank line before any output shown on the screen:

trap 'printf "\
"' DEBUG

You can also change the command prompt by exporting PS1 variables to make the output easier to read.
Command: To do this, run the following command:

export PS1="\
[\u@\h \W] $ "

Command: To configure the AWS CLI, run the following command:

aws configure

Command: When prompted, configure the following:

• AWS Access Key ID [None]: Press Enter
• AWS Secret Access Key [None]: Press Enter
• Default region name [None]: Copy and paste the Region value
• Default output format [None]: json

(Optional) Note: If the content received is Partial credentials found in shared-credentials-file, missing: aws_secret_access_key Missing: aws_secret_access_key) error message, please run the following command to delete the credential file and re-execute the aws configure command. Otherwise, continue to the next step.

rm ~/.aws/credentials

Command: To list the S3 buckets you have access to in your account, run the following command:

aws s3 ls

Command: To list all files in LabBucket, run the following command. Replace with the corresponding value for LabBucket:

aws s3 ls s3://<LabBucket>

labstack-2848353a-a5d2-4c70-b586-997e2b-labbucket-c72jm1b2ozlq
labstack-2848353a-a5d2-4c70-b586-labloggingbucket-1ctuk4k4ixe1

Command: To copy the file from Amazon S3 to your local home directory, run the following command. Replace with the corresponding value for LabBucket:

aws s3 cp s3://<LabBucket>/demo.txt ~/

labstack-2848353a-a5d2-4c70-b586-997e2b-labbucket-c72jm1b2ozlq

Command: To display the contents of this file to the screen, run the following command:

less demo.txt

Ability to download files from Amazon S3 to a public instance. Type q to exit the editor. Next, modify the file from the home directory and re-upload it to LabBucket.

Command: To add text to the file and display the updated file contents on the screen, run the following command:

echo "
This is some non-unique text that will be appended to your file." >> demo.txt

less demo.txt

Command: To re-upload the updated files back to Amazon S3, run the following command. Replace with the corresponding value for LabBucket:

aws s3 cp demo.txt s3://<LabBucket>/

labstack-2848353a-a5d2-4c70-b586-997e2b-labbucket-c72jm1b2ozlq

Switch to a browser tab that displays the AWS Management Console. In the bucket list, select the bucket name that matches the LabBucket value on the left side of the experiment page. Select the name of the demo.txt file. Select Open. The file will open in a new browser tab. The file was successfully updated from the public instance. Note: If the file does not open in a browser tab, make sure pop-ups are not blocked.

![image-20230831171133573](https://img-blog.csdnimg.cn/img_convert/bcc93fda9122a095626892183e1ce533.png

Next, run the same command against an instance located in a private subnet that does not have a route to the internet. Amazon S3 is not in a VPC; without a route to the internet, the bucket will not be accessible. This also means that all requests to Amazon S3 from a public instance will need a route to the Internet to reach the public Amazon S3 endpoint.

Command: To access your home directory, run the following command:

cd ~ & amp; & amp; trap 'printf "\
"' DEBUG & amp; & amp; export PS1="\
[\u@\h \W] $ " & amp; & aws configure

The command times out after approximately 5 minutes. Wait about five minutes and press ctrl + c to interrupt the process to avoid a timeout. Private instances have no route to Amazon S3 because there is no internet gateway, NAT gateway, or VPC endpoint.

Task 3: Create a VPC endpoint for Amazon S3

In this task, you create a VPC gateway endpoint to access Amazon S3 from a private instance. You can use the AWS CLI to run commands and create endpoints. To enable the AWS CLI to communicate with resources outside your VPC (that is, global AWS services), you need to run commands from a public instance.

The following figure shows all the resources provisioned for this lab task and how they are connected:

Switch back to the tab used to connect to the public instance.

To list the services for which VPC endpoints have been created, run the following command:

aws ec2 describe-vpc-endpoints --query 'VpcEndpoints[*].ServiceName'

Command: To find the VPC ID for labVPC, run the following command:

VPC=$(aws ec2 describe-vpcs --query 'Vpcs[*].VpcId' --filters 'Name=tag:Name, Values=labVPC' | jq -r '.[0]')

echo $VPC

This command uses JQ to process the JSON output and format it correctly for later use.

Command: To find the route table ID of a private route table, run the following command:

RTB=$(aws ec2 describe-route-tables --query 'RouteTables[*].RouteTableId' --filters 'Name=tag:Name, Values=PrivateRouteTable' | jq -r '.[0]')

echo $RTB

?Command?: To create an Amazon S3 endpoint, run the following command:

export AWS_REGION=$(curl -s 169.254.169.254/latest/dynamic/instance-identity/document | jq -r '.region')

echo $AWS_REGION

aws ec2 create-vpc-endpoint \
    --vpc-id $VPC \
    --service-name com.amazonaws.$AWS_REGION.s3 \
    --route-table-ids $RTB

Note: Backslashes have been added to the above command to run as a multi-line command.

The Amazon S3 VPC endpoint is now created.

Command: To verify that the Amazon S3 VPC endpoint was created, run the following command:

aws ec2 describe-vpc-endpoints --query 'VpcEndpoints[*].ServiceName'

The Amazon S3 endpoint now appears in the list.

Task 4: Interacting with Amazon S3 through a private instance

In this task, you will use an instance located in a private subnet that did not previously have access to Amazon S3. Now that the Amazon S3 endpoint is created and associated with the route table associated with the private subnet, you can access Amazon S3 directly without using public resources, as shown in the following image:

Switch to the tab used to connect to the private instance.

To list the S3 buckets that you have access to in your account, run the following command:

aws s3 ls

Command: To list all files in LabBucket, run the following command. Replace with the corresponding value for LabBucket:

aws s3 ls s3://<LabBucket>

labstack-2848353a-a5d2-4c70-b586-997e2b-labbucket-c72jm1b2ozlq

This command will display a list of files that match the files in the S3 bucket you saw earlier from the console.

Command: To display a list of files in the instance home directory, run the following command:

cd ~
ls -l

Note that there are no local files in this directory because the files were previously downloaded from S3 to the public instance and are now connected to the private instance.

Command: To copy the file from Amazon S3 to your local home directory, run the following command. Replace with the corresponding value for LabBucket:

aws s3 cp s3://<LabBucket>/demo.txt ~/

labstack-2848353a-a5d2-4c70-b586-997e2b-labbucket-c72jm1b2ozlq

Command: To display the contents of this file to the screen, run the following command:

less demo.txt

Now that you have the Amazon S3 VPC endpoint, you can download files from Amazon S3 on your private instance.

Add VPC endpoint policy

VPC gateway endpoints allow you to specify policies that restrict access. Using endpoint policies, you can specify exactly which Amazon S3 buckets are allowed to be accessed by instances in a private subnet. For example, a policy allows access to the data bucket, but restricts access to the logging bucket.

Use the console or AWS CLI to add a policy to the Amazon S3 VPC gateway endpoint to allow access to labbucket but deny access to labloggingbucket< /strong>, as shown below:

{<!-- -->
    "Version": "2012-10-17",
    "Statement": [
        {<!-- -->
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:List*",
            "Resource": "arn:aws:s3:::*"
        },
        {<!-- -->
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<LabBucket>",
                "arn:aws:s3:::<LabBucket>/*"
            ]
        },
        {<!-- -->
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<LabLoggingBucket>",
                "arn:aws:s3:::<LabLoggingBucket>/*"
            ]
        }
    ]
}

{<!-- -->
    "Version": "2012-10-17",
    "Statement": [
        {<!-- -->
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:List*",
            "Resource": "arn:aws:s3:::*"
        },
        {<!-- -->
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<LabBucket>",
                "arn:aws:s3:::<LabBucket>/*"
            ]
        },
        {<!-- -->
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<LabLoggingBucket>",
                "arn:aws:s3:::<LabLoggingBucket>/*"
            ]
        }
    ]
}

aws s3 ls s3://<LabBucket>
aws s3 ls s3://<LabLoggingBucket>

cd ~
cat <> policy.json
{<!-- -->
    "Version": "2012-10-17",
    "Statement": [
        {<!-- -->
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:List*",
            "Resource": "arn:aws:s3:::*"
        },
        {<!-- -->
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<LabBucket>",
                "arn:aws:s3:::<LabBucket>/*"
            ]
        },
        {<!-- -->
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<LabLoggingBucket>",
                "arn:aws:s3:::<LabLoggingBucket>/*"
            ]
        }
    ]
}
EOT

export vpcEndpointId=$(aws ec2 describe-vpc-endpoints --query 'VpcEndpoints[?contains(ServiceName, `s3`) == `true`].VpcEndpointId' --output text)

echo ${vpcEndpointId}

aws ec2 modify-vpc-endpoint --vpc-endpoint-id ${vpcEndpointId} --policy-document file://policy.json

aws s3 ls s3://<LabBucket>
aws s3 ls s3://<LabLoggingBucket>