AWS Transfer – Public FTP

AWS Transfer supports 3 protocols: SFTP, FTP, and FTPS. And only SFTP can have a public endpoint, FTP/FTPS can only be run inside a VPC. Also for login/password authorization, you must use a custom provider, you can find more information about this here.

Goal:

Create an AWS Transfer server for the FTP protocol, the service must be public and authorization must also be by login / password.

FTP is insecure and AWS does not recommend using it on public networks.

 

The first thing you need is the AWS SAM CLI installed.

Create a directory where we will download the template, go to it and download:

wget https://s3.amazonaws.com/aws-transfer-resources/custom-idp-templates/aws-transfer-custom-idp-secrets-manager-sourceip-protocol-support-apig.zip

 

Unzip and run the following command:

sam deploy --guided --stack-name aws-transfer-ftp

 

Where, "aws-transfer-ftp" is the name of the created CloudFormation stack, if you specify the name of an existing one, it will update it.

 

Then the interactive installation will start, where you will be prompted to specify the following parameters:

  • Stack Name – the name of the CloudFormation stack, the default is the value of the "–stack-name" key parameter;
  • AWS Region – the region where the CloudFormation stack will be deployed;
  • Parameter CreateServer – whether AWS Transfer service will be created (by default – true);
  • Parameter SecretManagerRegion – if your region does not support SecretsManager, then you can specify a separate region for it;
  • Parameter TransferEndpointType – PUBLIC or VPC, since FTP does not support public endpoints, specify VPC;
  • Parameter TransferSubnetIDs – ID's of the subnets in which the AWS Transfer endpoint will be;
  • Parameter TransferVPCID – VPC ID where the subnets specified in the previous parameter are located.

 

 

Let’s create a SecurityGroup for the FTP service in the required VPC. And we will allow incoming traffic to TCP ports 21 and 81928200 from any address. While we save the created SG, we will attach it in the future.

Then go to the AWS Console – "AWS Transfer Family", find the server created by AWS Transfer and edit its protocol, uncheck the "SFTP" protocol and select "FTP" protocol, and save the changes.

Now we need to add access to FTP from the world, for this we will use NLB. First, let’s find out the private IP addresses of VPC endpoint for AWS Transfer, for this in the "Endpoint details" block, click on the link to the VPC endpoint.

 

Go to the "Subnets" tab and copy all the IP addresses, they will be needed to create target groups.

 

Go to the "Security Groups" tab and change the default security group to the one created earlier.

Now let’s create a target group, for this in the AWS console go to "EC2" -> "Load Balancing" -> "Target Groups" and create the first target group for TCP port 21.

  • Target type: IP address
  • Protocol: TCP
  • Port: 21

It is better to indicate the port number in the name of the target group at the end, since there will be 10 of them and you can easily get confused.

We will also indicate the VPC in which the AWS Transfer service was created. In the next tab, one by one, we will indicate the IP addresses of the VPC endpoint, which we looked at earlier. We save the target group.

Now you need to create 9 more target groups for the port range: TCP 81928200. The procedure is the same as for the target group for port 21, except that you need to specify port 21 for HeathCheck. To do this, in the "Health checks" block, open the "Advanced health check setting" tab, select "Overrive" and specify the port number – 21.

After we are done with target groups, we need to create an "internet-facing" Network Load Balancer and place it on public networks of the same VPC where the AWS Transfer service is. We also create 10 listeners, for TCP ports 21, and for the range 81928200, and for each listener we point the desired target group corresponding to the port number. After which the FTP service must be accessible from outside.

In order to add an FTP user, go to the "Secrets Manager" in the AWS console and create a secret with the "Other type of secrets" type.

 

Create 3 "key/value" pairs:

  • Password – password for the new FTP user;
  • Role – ARN of the role that has write permission to the required S3 bucket;
  • HomeDirectoryDetails – [{"Entry": "/", "Target": "/s3-bucket/user-name"}]

Where "s3-bucket" is the name of the S3 bucket, "user-name" is the name of the directory that the user will go to when connecting to the FTP server (the directory name does not have to match the username, and may also be located outside the root of the bucket)

 

We must save the secret with a name in the format: "server_id/user_name", where "server_id" is the AWS Transfer server ID, "user_name" is the username that will be used to connect to the FTP server.

For convenience, you can also create a DNS CNAME record for the NLB record.

FIX ERROR – RDS: Error creating DB Parameter Group: InvalidParameterValue: ParameterGroupFamily

When creating an RDS by specifying an incorrect value for the "ParameterGroupFamily" parameter, a similar error may occur:

Error creating DB Parameter Group: InvalidParameterValue: ParameterGroupFamily default.mariadb10.2 is not a valid parameter group family

 

To see a list of all possible values for the "ParameterGroupFamily" parameter, you can use the following command:

aws rds describe-db-engine-versions --query "DBEngineVersions[].DBParameterGroupFamily"

 

 

Nginx – Regular Expression Tester

 

For quick testing of Nginx regular expressions, you can use a ready-made docker image. To do this, you need to clone the NGINX-Demos repository:

git clone https://github.com/nginxinc/NGINX-Demos

 

Follow to the "nginx-regex-tester" directory:

cd NGINX-Demos/nginx-regex-tester/

 

And launch the container using "docker-compose":

docker-compose up -d

 

And open the next page:

http://localhost/regextester.php

 

AWS – EKS Fargate – Fluentd CloudWatch

At the time of writing, EKS Fargate does not support a driver log for recording to CloudWatch. The only option is to use Sidecar

Let’s create a ConfigMap, in which we indicate the name of the EKS cluster, region and namespace:

kubectl create configmap cluster-info \
--from-literal=cluster.name=YOUR_EKS_CLUSTER_NAME \
--from-literal=logs.region=YOUR_EKS_CLUSTER_REGION -n KUBERNETES_NAMESPACE

 

Next, let’s create a service account and a ConfigMap with a configuration file for Fluentd. To do this, copy the text below and save it as "fluentd.yaml"

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: {{NAMESPACE}}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd-role
rules:
  - apiGroups: [""]
    resources:
      - namespaces
      - pods
      - pods/logs
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluentd-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluentd-role
subjects:
  - kind: ServiceAccount
    name: fluentd
    namespace: {{NAMESPACE}}
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: {{NAMESPACE}}
  labels:
    k8s-app: fluentd-cloudwatch
data:
  fluent.conf: |
    @include containers.conf

    <match fluent.**>
      @type null
    </match>
  containers.conf: |
    <source>
      @type tail
      @id in_tail_container_logs
      @label @containers
      path /var/log/application.log
      pos_file /var/log/fluentd-containers.log.pos
      tag *
      read_from_head true
      <parse>
        @type none
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      </parse>
    </source>

    <label @containers>
      <filter **>
        @type kubernetes_metadata
        @id filter_kube_metadata
      </filter>

      <filter **>
        @type record_transformer
        @id filter_containers_stream_transformer
        <record>
          stream_name "#{ENV.fetch('HOSTNAME')}"
        </record>
      </filter>

      <filter **>
        @type concat
        key log
        multiline_start_regexp /^\S/
        separator ""
        flush_interval 5
        timeout_label @NORMAL
      </filter>

      <match **>
        @type relabel
        @label @NORMAL
      </match>
    </label>

    <label @NORMAL>
      <match **>
        @type cloudwatch_logs
        @id out_cloudwatch_logs_containers
        region "#{ENV.fetch('REGION')}"
        log_group_name "/aws/containerinsights/#{ENV.fetch('CLUSTER_NAME')}/application"
        log_stream_name_key stream_name
        remove_log_stream_name_key true
        auto_create_stream true
        <buffer>
          flush_interval 5
          chunk_limit_size 2m
          queued_chunks_limit_size 32
          retry_forever true
        </buffer>
      </match>
    </label>

 

And apply it:

curl fluentd.yaml | sed "s/{{NAMESPACE}}/default/" | kubectl apply -f -

 

Where "default" is the name of the required namespace

 

An example of a sidecar deployment:

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: testapp
  name: testapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: testapp
  strategy: {}
  template:
    metadata:
      labels:
        app: testapp
    spec:
      serviceAccountName: fluentd
      terminationGracePeriodSeconds: 30
      initContainers:
        - name: copy-fluentd-config
          image: busybox
          command: ['sh', '-c', 'cp /config-volume/..data/* /fluentd/etc']
          volumeMounts:
            - name: config-volume
              mountPath: /config-volume
            - name: fluentdconf
              mountPath: /fluentd/etc
      containers:
      - image: alpine:3.10
        name: alpine
        command: ["/bin/sh"]
        args: ["-c", "while true; do echo hello 2>&1 | tee -a /var/log/application.log; sleep 10;done"]
        volumeMounts:
        - name: fluentdconf
          mountPath: /fluentd/etc
        - name: varlog
          mountPath: /var/log
      - image: fluent/fluentd-kubernetes-daemonset:v1.7.3-debian-cloudwatch-1.0
        name: fluentd-cloudwatch
        env:
          - name: REGION
            valueFrom:
              configMapKeyRef:
                name: cluster-info
                key: logs.region
          - name: CLUSTER_NAME
            valueFrom:
              configMapKeyRef:
                name: cluster-info
                key: cluster.name
          - name: AWS_ACCESS_KEY_ID
            value: "XXXXXXXXXXXXXXX"
          - name: "AWS_SECRET_ACCESS_KEY"
            value: "YYYYYYYYYYYYYYY"
        resources:
          limits:
            memory: 400Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
          - name: config-volume
            mountPath: /config-volume
          - name: fluentdconf
            mountPath: /fluentd/etc
          - name: varlog
            mountPath: /var/log
      volumes:
        - name: config-volume
          configMap:
            name: fluentd-config
        - name: fluentdconf
          emptyDir: {}
        - name: varlog
          emptyDir: {}

 

In this deployment, the variables are set to "AWS_ACCESS_KEY_ID" and "AWS_SECRET_ACCESS_KEY", since at the moment there are endpoints for IAM roles only for services: EC2, ECS Fargate and Lambda. For avoid this you can use OpenID Connect provider for EKS.

Kubernetes – One role for multiple namespaces

 

 

Goal:

There are 2 namespaces, they are "kube-system" and "default". It is necessary to run a cron task in the "kube-system" namespace, which will clear the executed jobs and pods in the "default" space. To do this, create a service account in the "kube-system" namespace, a role with the necessary rights in the "default" namespace, and bind the created role for the created account.

cross-namespace-role.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jobs-cleanup
  namespace: kube-system
automountServiceAccountToken: false
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: jobs-cleanup
  namespace: default
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list", "delete"]
- apiGroups: ["batch", "extensions"]
  resources: ["jobs"]
  verbs: ["get", "list", "watch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jobs-cleanup
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jobs-cleanup
subjects:
- kind: ServiceAccount
  name: jobs-cleanup
  namespace: kube-system

 

 

FIX ERROR – EKS: cloudwatch – 0/1 nodes are available: 1 Insufficient pods.

After installing CloudWatch Agent in the EKS cluster, its pods stuck in the "Pending" state

 

Watching the describe pod

 

Solution:

The solution was found here. In this EKS cluster, a separate NodeGroup is allocated for the system pods with the instance type: t3.micro, and there is simply not enough capacity to launch the CloudWatch Agent.

 

After changing the type of instance upwards – t3.small, all pods switched to the "Running" status

EKS has limits on the number of pods per node, this limit can be calculated using the following formula:

N * (M-1) + 2

Where, N is the number of Elastic Network Interfaces (ENI) for this type of instance, M is the number of IP addresses for one ENI

The N and M values for a specific instance can be found here.

FIX ERROR – fdisk: DOS partition table format cannot be used on drives for volumes larger than 2199023255040 bytes for 512-byte sectors

When trying to create a partition with the "fdisk" utility, 2 or more TB in size, the following message appears:

The size of this disk is 2 TiB (2199023255552 bytes). DOS partition table format cannot be used on drives for volumes larger than 2199023255040 bytes for 512-byte sectors. Use GUID partition table format (GPT).

 

Solution:

Add the "gpt" label on the new disk

parted /dev/nvme2n1

 

Where "nvme2n1" – is your disk

 

Add the label:

mklabel gpt

 

Checking:

print

 

 

Then we return to fdisk and create the desired partition. The section can also be created in the "parted" utility

 

Let’s create a file system, for this we copy the full path of the created partition

 

And we create a file system, in this example it’s ext4:

mkfs.ext4 /dev/nvme2n1p1

 

Jenkins – Active Choice: GitHub – Commit

 

For a parameterized assembly with an image tag selection, you will need the Active Choices plugin

Go to "Manage Jenkins"

 

Section "Manage Plugins"

 

Go to the "Available" tab and select "Active Choices" in the search.

Install it.

Create a "New Item" – "Pipeline", indicate that it will be a parameterized assembly, and add the parameter "Active Choices Reactive Parameter"

 

We indicate that this is "Groovy Script" and paste the following into it:

import jenkins.model.*
import groovy.json.JsonSlurper

credentialsId = 'artem-github'
gitUri = '[email protected]:artem-gatchenko/ansible-openvpn-centos-7.git'

def creds = com.cloudbees.plugins.credentials.CredentialsProvider.lookupCredentials(
  com.cloudbees.plugins.credentials.common.StandardUsernameCredentials.class, Jenkins.instance, null, null ).find{
    it.id == credentialsId}

def slurper = new JsonSlurper()

def account = gitUri.split(":")[-1].split("/")[0]
def repo = gitUri.split(":")[-1].split("/")[-1].split("\\.")[0]

def addr = "https://api.github.com/repos/${account}/${repo}/commits"
def authString = "${creds.username}:${creds.password}".getBytes().encodeBase64().toString()
def conn = addr.toURL().openConnection()
conn.setRequestProperty( "Authorization", "Basic ${authString}" )
def response_json = "${conn.content.text}"
def parsed = slurper.parseText(response_json)
def commit = []
commit.add("Latest")
for (int i = 0; i < parsed.size(); i++) {
    commit.add(parsed.get(i).sha)
}
return commit

 

Where the value of variables, "credentialsId" – Jenkins Credentials ID with token to GitHub;

"gitUri" – the full path to the desired repository;

 

 

The same thing, but already in Pipeline

Pipeline:

properties([
  parameters([
    [$class: 'StringParameterDefinition',
      defaultValue: '[email protected]:artem-gatchenko/ansible-openvpn-centos-7.git',
      description: 'Git repository URI',
      name: 'gitUri',
      trim: true
    ],
    [$class: 'CascadeChoiceParameter', 
      choiceType: 'PT_SINGLE_SELECT', 
      description: 'Select Image',
      filterLength: 1,
      filterable: false,
      referencedParameters: 'GIT_URI',
      name: 'GIT_COMMIT_ID', 
      script: [
        $class: 'GroovyScript', 
        script: [
          classpath: [], 
          sandbox: false, 
          script: 
            '''
            import jenkins.model.*
            import groovy.json.JsonSlurper

            credentialsId = 'artem-github'

            def creds = com.cloudbees.plugins.credentials.CredentialsProvider.lookupCredentials(
              com.cloudbees.plugins.credentials.common.StandardUsernameCredentials.class, Jenkins.instance, null, null ).find{
                it.id == credentialsId}

            def slurper = new JsonSlurper()

            def account = gitUri.split(":")[-1].split("/")[0]
            def repo = gitUri.split(":")[-1].split("/")[-1].split("\\\\.")[0]

            def addr = "https://api.github.com/repos/${account}/${repo}/commits"
            def authString = "${creds.username}:${creds.password}".getBytes().encodeBase64().toString()
            def conn = addr.toURL().openConnection()
            conn.setRequestProperty( "Authorization", "Basic ${authString}" )
            def response_json = "${conn.content.text}"
            def parsed = slurper.parseText(response_json)
            def commit = []
            commit.add("Latest")
            for (int i = 0; i < parsed.size(); i++) {
                commit.add(parsed.get(i).sha)
            }
            return commit
            '''
        ]
      ]
    ]
  ])
])