AWS – S3: Allow public access to objects over VPN

Goal:

Allow public read access for all objects in the S3 bucket only using a VPN connection, objects must be non-public to connect from the world. OpenVPN is used as a VPN service, which can be deployed anywhere, so we will build an allow a rule to check the IP address.

 

First you need to find out the list of networks that belong to the endpoints of the S3 service in the region we need, so as not to wrap all traffic through the VPN. To do this, download the current list of networks and parse it:

jq '.prefixes[] | select(.region=="eu-central-1") | select(.service=="S3") | .ip_prefix' < ip-ranges.json

 

Where, "eu-central-1" is the region where the necessary S3 bucket is located.

 

You should get an output like:

"52.219.170.0/23"
"52.219.168.0/24"
"3.5.136.0/22"
"52.219.72.0/22"
"52.219.44.0/22"
"52.219.169.0/24"
"52.219.140.0/24"
"54.231.192.0/20"
"3.5.134.0/23"
"3.65.246.0/28"
"3.65.246.16/28"

 

Now we translate the subnet mask into a 4-byte format and add the parameters to the OpenVPN server configuration as "push" parameters:

push "route 52.219.170.0 255.255.254.0"
push "route 52.219.168.0 255.255.255.0"
push "route 3.5.136.0 255.255.252.0"
push "route 52.219.72.0 255.255.252.0"
push "route 52.219.44.0 255.255.252.0"
push "route 52.219.169.0 255.255.255.0"
push "route 52.219.140.0 255.255.255.0"
push "route 54.231.192.0 255.255.240.0"
push "route 3.5.134.0 255.255.254.0"
push "route 3.65.246.0 255.255.255.240"
push "route 3.65.246.16 255.255.255.240"

 

We restart the OpenVPN server service and after reconnecting we should get a list of required networks and traffic that will go through the VPN connection.

 

Now it remains to add the following policy to the S3 bucket:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Allow only from VPN",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::artem-services",
                "arn:aws:s3:::artem-services/*"
            ],
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": "1.2.3.4"
                }
            }
        }
    ]
}

 

Where, "artem-services" is the name of the S3 bucket and "1.2.3.4" is the IP address of the OpenVPN server.

 Python – AWS EBS creating snapshots based on a tag and keeping only one latest version

This script looks for an EBS in the region "eu-west-1" with a tag whose key is "Application" and the value is passed as an argument, creating a snapshot of this EBS. In the same way, it searches for a snapshot by tag and deletes everything except the last one.

An example of running to create a snapshot:

python3 main.py create artem-test-app

 

An example of running to delete old snapshots:

python3 main.py cleanup artem-test-app

 

main.py:

from __future__ import print_function
from datetime import datetime
import sys
import boto3
import botocore
import urllib.request

ec2 = boto3.client('ec2', region_name='eu-west-1')

def create_snapshot(app):
    print("Creating snapshot for " + str(app))
    # Get Instance ID
    instances_id = []
    response = ec2.describe_instances(
        Filters = [
            {
                'Name': 'tag:Application',
                'Values': [
                    app,
                ]
            },
            {
                'Name': 'instance-state-name',
                'Values': ['running']
            }
        ]
    )

    for reservation in response['Reservations']:
        for instance in reservation['Instances']:
            instances_id.append(instance['InstanceId'])

    # Get Volumes ID
    volumes_id = []
    for instance_id in instances_id:
        response = ec2.describe_instance_attribute(InstanceId = instance_id, Attribute = 'blockDeviceMapping')
        for BlockDeviceMappings in response['BlockDeviceMappings']:
            volumes_id.append(BlockDeviceMappings['Ebs']['VolumeId'])

    # Create Volume Snapshot
    for volume_id in volumes_id:
        date = datetime.now()
        response = ec2.create_snapshot(
            Description = app + " snapshot created by Jenkins at " + str(date),
            VolumeId = volume_id,
            TagSpecifications=[
                {
                    'ResourceType': 'snapshot',
                    'Tags': [
                        {
                            'Key': 'Name',
                            'Value': app + "-snapshot"
                        },
                    ]
                },
            ]
        )


        try:
            print("Waiting until snapshot will be created...")
            snapshot_id = response['SnapshotId']
            snapshot_complete_waiter = ec2.get_waiter('snapshot_completed')
            snapshot_complete_waiter.wait(SnapshotIds=[snapshot_id], WaiterConfig = { 'Delay': 30, 'MaxAttempts': 120})

        except botocore.exceptions.WaiterError as e:
                print(e)

def cleanup_snapshot(app):
    print("Clean up step.")
    print("Looking for all snapshots regarding " + str(app) + " application.")
    # Get snapshots
    response = ec2.describe_snapshots(
        Filters = [
            {
                'Name': 'tag:Name',
                'Values': [
                    app + "-snapshot",
                ]
            },
        ],
        OwnerIds = [
            'self',
        ]
    )

    sorted_snapshots = sorted(response['Snapshots'], key=lambda k: k['StartTime'])

    # Clean up snapshots and keep only the latest snapshot
    for snapshot in sorted_snapshots[:-1]:
        print("Deleting snapshot: " + str(snapshot['SnapshotId']))
        response = ec2.delete_snapshot(
            SnapshotId = snapshot['SnapshotId']
        )

def main():
    if sys.argv[1:] and sys.argv[2:]:
        action = sys.argv[1]
        app = sys.argv[2]
        if action == 'create':
            create_snapshot(app)
        elif action == 'cleanup':
            cleanup_snapshot(app)
        else:
            print("Wrong action: " + str(action))
    else:
        print("This script for create and cleanup snapshots")
        print("Usage  : python3 " + sys.argv[0] + " {create|cleanup} " + "{application_tag}")
        print("Example: python3 " + sys.argv[0] + " create " + "ebs-snapshot-check")

if __name__ == '__main__':
    main()

 Jenkins – Active Choice: SSH – Return remote command values

For a parameterized assembly with an image tag selection, you will need the Active Choices plugin

Go to "Manage Jenkins"

 

Section "Manage Plugins"

 

Go to the "Available" tab and select "Active Choices" in the search.

Install it.

 

Create a "New Item" – "Pipeline", indicate that it will be a parameterized assembly, and add the parameter "Active Choices Parameter"

 

 

 

We indicate that this is "Groovy Script" and paste the following into it:

import com.cloudbees.plugins.credentials.Credentials;
import com.cloudbees.plugins.credentials.CredentialsNameProvider;
import com.cloudbees.plugins.credentials.common.StandardCredentials;
import com.cloudbees.jenkins.plugins.sshcredentials.SSHUserPrivateKey;
import com.jcraft.jsch.JSch;
import com.jcraft.jsch.Session;
import com.jcraft.jsch.ChannelExec;
import jenkins.model.*


sshCredentialsId = 'instance_ssh_key'
sshUser = 'artem'
sshInstance = '192.168.1.100'

def sshCreds = com.cloudbees.plugins.credentials.CredentialsProvider.lookupCredentials(SSHUserPrivateKey.class, Jenkins.instance, null, null ).find({it.id == sshCredentialsId});

String ssh_key_data = sshCreds.getPrivateKeys()

JSch jsch = new JSch();
jsch.addIdentity("id_rsa", ssh_key_data.getBytes(), null, null);
Session session = jsch.getSession(sshUser, sshInstance, 22);
Properties prop = new Properties();
prop.put("StrictHostKeyChecking", "no");
session.setConfig(prop);

session.connect();

ChannelExec channelssh = (ChannelExec)session.openChannel("exec");
channelssh.setCommand("cat /home/artem/secret_list");
channelssh.connect();

def result = []
InputStream is=channelssh.getInputStream();
is.eachLine {
    result.add(it)
}
channelssh.disconnect();

return result

 

Where is the value of the variables, "sshCredentialsId" – Jenkins Credentials ID with type: "SSH Username with private key";

"sshUser" – username for SSH connection;

"sshInstance" – IP address or domain name of the remote instance;

"cat /home/artem/secret_list" – the command whose result we want to return

 

 Jenkins – Active Choice: S3 – Return necessary key value from JSON file

For a parameterized assembly with an image tag selection, you will need the Active Choices plugin

Go to "Manage Jenkins"

 

Section "Manage Plugins"

 

Go to the "Available" tab and select "Active Choices" in the search.

Install it.

 

Create a "New Item" – "Pipeline", indicate that it will be a parameterized assembly, and add the parameter "Active Choices Parameter"

 

 

 

We indicate that this is "Groovy Script" and paste the following into it:

import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.S3Object;
import com.cloudbees.plugins.credentials.Credentials;
import com.cloudbees.plugins.credentials.CredentialsNameProvider;
import com.cloudbees.plugins.credentials.common.StandardCredentials;
import com.cloudbees.jenkins.plugins.awscredentials.AmazonWebServicesCredentials;
import groovy.json.JsonSlurper;
import java.nio.charset.Charset;
import java.nio.charset.StandardCharsets;
import java.util.stream.Collectors;
import jenkins.model.*
  
def AWS_REGION = 'eu-west-1'
def BUCKET_NAME = 'artem.services'
def INFO_FILE = 'info.json'

credentialsId = 'aws_credentials'

def creds = com.cloudbees.plugins.credentials.CredentialsProvider.lookupCredentials(AmazonWebServicesCredentials.class, Jenkins.instance, null, null ).find({it.id == credentialsId});


AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withCredentials(creds).withRegion(AWS_REGION).build(); 

S3Object s3Object = s3Client.getObject(BUCKET_NAME, INFO_FILE)

try {
    BufferedReader reader = new BufferedReader(new InputStreamReader(s3Object.getObjectContent()))
    s3Data = reader.lines().collect(Collectors.joining("\n"));
} catch (Exception e) {
    System.err.println(e.getMessage());
    System.exit(1);
}

def json = new JsonSlurper().parseText(s3Data)

return json.instance.ip

 

Where is the value of the variables, "credentialsId" – Jenkins Credentials ID with AWS Access Key and AWS Secret Key (if the Jenkins instance does not use IAM Role, or it is not deployed in AWS Cloud);

"AWS_REGION" – AWS region where S3 Bucket is located;

"BUCKET_NAME" – S3 bucket name;

"INFO_FILE" – the key of the object in the bucket, in our case a JSON file;

"return json.instance.ip" – return key value instance: {'ip': "1.2.3.4}

 

 Ansible – Store N recent directories and artifacts

Goal:

There is a directory "/opt/application" where the archive with the application is downloaded and unpacked into a directory where the short derivative of the HASH commit (8 characters) is used as the name. And a symbolic link is created to this directory. It is necessary to store only the 3 latest versions of the application, both directories, and archives, and also do not delete directories called:

  • logs
  • media

Also, the deletion should be done using regex, so that in the future it does not delete directories that may be created.

Playbook:

---
- hosts: all
  tasks:
    - name: Find all application directories
      find:
        paths: "/opt/application/"
        file_type: directory
        use_regex: yes
        patterns:
          - '[a-z0-9]{8,8}'
        excludes: 'logs,media'
      register: dirs

    - name: Keep only the last 3 directories
      file:
        path: "{{ item }}"
        state: absent
      with_items: "{{ (dirs.files | sort(attribute='atime', reverse=True))[3:] | map(attribute='path') | list }}"

    - name: Find application artifacts
      find:
        paths: "/opt/application/"
        file_type: file
        use_regex: yes
        patterns:
          - '^[a-z0-9]{8,8}.tar.gz$'
      register: artifacts

    - name: Keep only the last 3 artifacts
      file:
        path: "{{ item }}"
        state: absent
      with_items: "{{ (artifacts.files | sort(attribute='atime', reverse=True))[3:] | map(attribute='path') | list }}"

 

 

 Python – AWS S3 keep N latest artifacts

This script gets a list of all directories in the bucket and deletes all objects in each directory, except for the last "N" specified. To run the script, you need to pass two arguments:

  • Bucket name
  • Number of last stored objects

How to use it:

python3 main.py artifacts-artem-services 3

 

main.py:

import sys
import boto3

def cleanup():
    get_last_modified = lambda obj: int(obj['LastModified'].strftime('%s'))

    s3 = boto3.client('s3')
    result = s3.list_objects(Bucket=bucket, Delimiter='/')
    for dir in result.get('CommonPrefixes'):
        print('Directory: ' + str(dir['Prefix']))
        artifacts_listing = s3.list_objects_v2(Bucket = bucket, Prefix = dir.get('Prefix'))['Contents']
        artifacts_sorted = [obj['Key'] for obj in sorted(artifacts_listing, key=get_last_modified)]
        for artifact in artifacts_sorted[:-keep_last]:
            print('Deleting artifact: ' + str(artifact))
            s3.delete_object(Bucket = bucket, Key = artifact)

if sys.argv[1:] and sys.argv[2:]:
    bucket = sys.argv[1]
    keep_last = int(sys.argv[2])
    cleanup()
else:
    print("This script for cleanup old artifacts in S3 bucket")
    print("Usage  : python3 " + sys.argv[0] + " {BUCKET_NAME} " + "{NUMBER_OF_THE_LAST_KEEPING_ARTIFACTS}")
    print("Example: python3 " + sys.argv[0] + " artifacts-artem-services " + "3")

 EKS – Encrypt current PV (EBS Volume)

The answer was taken from gitmemory

In order to encrypt an already created EBS Volume, you need to take a snapshot of it. Then, from the created snapshot, create a disk in the same region as the original one, and also specify the KMS key for encryption.

Then we save the manifest of the current PV to a file:

kubectl get pv <PV_NAME> -o yaml > /tmp/pv.yaml

 

We edit the file, replacing the ID of the original disk with the encrypted one.

Then apply the changes:

kubectl replace --cascade=false --force -f /tmp/pv.yaml

 

The previous command will "get stuck" on execution, as the "finalizers" parameter prevents it, so in the next tab we do the following:

kubectl edit pv <PV_NAME>

 

Find and remove the following:

  finalizers:
  - kubernetes.io/pv-protection

 

We save the changes, after which the command in the previous tab should work successfully.

 

After that, patch the PVC to which this PV belongs:

kubectl patch pvc <PVC_NAME> -p '{"metadata":{"finalizers": []}}' --type=merge

 

Now all that’s left is to delete the pod that the PV is mounted to and make sure it is re-created with the new PV mounted. Also, do not forget about the rights to use KMS keys for the IAM role, which is attached to EKS nodes.

 Jenkins – Active Choice: PostgreSQL – Return result of SELECT query

For a parameterized assembly with an image tag selection, you will need the Active Choices plugin

Go to "Manage Jenkins"

 

Section "Manage Plugins"

 

Go to the "Available" tab and select "Active Choices" in the search.

Install it. You also need plugins:

Create a "New Item" – "Pipeline", indicate that it will be a parameterized assembly, and add the parameter "Active Choices Parameter"

 

 

 

We indicate that this is "Groovy Script" and paste the following into it:

import groovy.sql.Sql
import java.sql.Driver

credentialsId = 'artem-services-rds-credentials'
url = 'artem-services-rds.xxxxxxxxxxxx.eu-central-1.rds.amazonaws.com:5432/postgres'

def creds = com.cloudbees.plugins.credentials.CredentialsProvider.lookupCredentials(
  com.cloudbees.plugins.credentials.common.StandardUsernameCredentials.class, Jenkins.instance, null, null ).find{
    it.id == credentialsId
  }

def driver = Class.forName('org.postgresql.Driver').newInstance() as Driver 

def props = new Properties()
props.setProperty("user", "${creds.username}") 
props.setProperty("password", "${creds.password}")

def conn = driver.connect("jdbc:postgresql://${url}", props) 
def sql = new Sql(conn)

def artifact = []

artifact.add("Not selected")

def rows = sql.rows("select * from users").each { row ->
  artifact.add("$row.first_name")
}

return artifact

 

Where is the value of the variables, "credentialsId" – Jenkins Credentials ID with login and password to connect to the database;

"url" – database connection string (endpoint + port + db name);

 

This Active Choice makes a SELECT query on the "users" table and returns only the values of the "first_name" fields, as well as adding "Not selected" to the first position of the result.

AWS – S3 Allow Access for Organization Members

In order to allow read access from the S3 Bucket for all members included in the organization, the following policy must be applied to the S3 Bucket:

{
  "Version": "2012-10-17",
  "Statement": {
    "Sid": "AllowOrganizationToReadBucket",
    "Effect": "Allow",
    "Principal": "*",
    "Action": [
      "s3:GetObject",
      "s3:ListBucket"
    ],
    "Resource": [
      "arn:aws:s3:::stackset-lambdas",
      "arn:aws:s3:::stackset-lambdas/*"
    ],
    "Condition": {
      "StringEquals": {"aws:PrincipalOrgID":["o-xxxxxxxxxx"]}
    }
  }
}

 

Where "stackset-lambdas" is the S3 Bucket name and "o-xxxxxxxxxx" is your Organization ID.