FIX ERROR – CentOS+Nginx+Jenkins: 502 Bad Gateway

When running Nginx as a reverse proxy to Jenkins on CentOS, a 502 error may occur. Nginx error log will be as follows:

2020/05/07 13:32:33 [crit] 9665#9665: *1 connect() to 127.0.0.1:8080 failed (13: Permission denied) while connecting to upstream, client: 1.2.3.4, server: jenkins.artem.services, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "jenkins.artem.services"

 

Solution:

Reason SELinux. You can either allow Jenkins or disable SELinux.

Option to disable SELinux

Open the configuration file:

vim /etc/selinux/config

 

And change "enforcing" to "disabled"

SELINUX=disabled

 

And restart the server.

In CentOS 6, you can disable SELinux without rebooting (CentOS 7/8 – require a reboot) by running the following command:

echo 0 > /selinux/enforce

Linux – Mount partition by label

List of labels

You can view the list of existing labels in the following path:

/dev/disk/by-label/

 

If this directory does not exist, then there are no labels in the system

 

Adding a label

For different file systems, the label is added differently.

ext2/ext3/ext4:

e2label /dev/sda1 LABEL

reiserfs:

reiserfstune -l LABEL /dev/sda1

jfs:

jfs_tune -L LABEL /dev/sda1

xfs:

xfs_admin -L LABEL /dev/sda1

 

Where, "LABEL" – a unique label, "/dev/sda1" – necessary partition

 

fstab

An example of recording in fstab to mount an xfs section on the "SITE" label

LABEL=SITE	/var/www/html/site	xfs	defaults	0 0

 

Jenkins – Active Choice: AWS ECR Images tag (AWS SDK)

 

For a parameterized assembly with an image tag selection, you will need the Active Choices plugin

Go to "Manage Jenkins"

 

Section "Manage Plugins"

 

Go to the "Available" tab and select "Active Choices" in the search.

Install it.

Create a "New Item" – "Pipeline", indicate that it will be a parameterized assembly, and add the parameter "Active Choices Reactive Parameter"

 

We indicate that this is "Groovy Script" and paste the following into it:

import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.services.ecr.AmazonECR;
import com.amazonaws.services.ecr.AbstractAmazonECR;
import com.amazonaws.services.ecr.AmazonECRClient;
import com.amazonaws.services.ecr.model.ListImagesRequest;
import com.amazonaws.services.ecr.model.ListImagesResult;
import com.amazonaws.services.ecr.AmazonECRClientBuilder;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.RegionUtils;
import com.amazonaws.regions.Regions;
import jenkins.model.*

AmazonECR client = AmazonECRClientBuilder.standard().withRegion("eu-west-1").build();
ListImagesRequest request = new ListImagesRequest().withRepositoryName("artem-services");
res = client.listImages(request);


def result = []
for (image in res) {
   result.add(image.getImageIds());
}

return result[0].imageTag;

 

Where, "eu-west-1" – region where the ECR repository is located;

"artem-services" – the name of your ECR repository;

 

 

The same thing, but already in Pipeline

Pipeline:

properties([
  parameters([
    [$class: 'CascadeChoiceParameter', 
      choiceType: 'PT_SINGLE_SELECT', 
      description: 'Select Image',
      filterLength: 1,
      filterable: false,
      name: 'ImageTag', 
      script: [
        $class: 'GroovyScript', 
        script: [
          classpath: [], 
          sandbox: false, 
          script: 
            '''
            import com.amazonaws.client.builder.AwsClientBuilder;
            import com.amazonaws.services.ecr.AmazonECR;
            import com.amazonaws.services.ecr.AbstractAmazonECR;
            import com.amazonaws.services.ecr.AmazonECRClient;
            import com.amazonaws.services.ecr.model.ListImagesRequest;
            import com.amazonaws.services.ecr.model.ListImagesResult;
            import com.amazonaws.services.ecr.AmazonECRClientBuilder;
            import com.amazonaws.regions.Region;
            import com.amazonaws.regions.RegionUtils;
            import com.amazonaws.regions.Regions;
            import jenkins.model.*

            AmazonECR client = AmazonECRClientBuilder.standard().withRegion("eu-west-1").build();
            ListImagesRequest request = new ListImagesRequest().withRepositoryName("artem-services");
            res = client.listImages(request);


            def result = []
            for (image in res) {
               result.add(image.getImageIds());
            }

            return result[0].imageTag;
            '''
        ]
      ]
    ]
  ])
])

Jenkins – Active Choice: AWS ECR Images tag (AWS Cli)

 

For a parameterized assembly with an image tag selection, you will need the Active Choices plugin

Go to "Manage Jenkins"

 

Section "Manage Plugins"

 

Go to the "Available" tab and select "Active Choices" in the search.

Install it.

Create a "New Item" – "Pipeline", indicate that it will be a parameterized assembly, and add the parameter "Active Choices Reactive Parameter"

 

We indicate that this is "Groovy Script" and paste the following into it:

def command = ['/bin/sh', '-c', '/var/lib/jenkins/.local/bin/aws ecr describe-images --region eu-west-1 --repository-name artem-services --query "sort_by(imageDetails,& imagePushedAt)[ * ].imageTags[ * ]" --output text']
  def proc = command.execute()
  return proc.text.readLines()

 

Where, "/var/lib/jenkins/.local/bin/aws" – is the full path to AWS Cli;

"eu-west-1" – region where the ECR repository is located;

"artem-services" – The name of your ECR repository;

 

 

The same thing, but already in Pipeline

Pipeline:

properties([
  parameters([
    [$class: 'CascadeChoiceParameter', 
      choiceType: 'PT_SINGLE_SELECT', 
      description: 'Select Image',
      filterLength: 1,
      filterable: false,
      name: 'ImageTag', 
      script: [
        $class: 'GroovyScript', 
        script: [
          classpath: [], 
          sandbox: false, 
          script: 
            '''
            def command = ['/bin/sh', '-c', '/var/lib/jenkins/.local/bin/aws ecr describe-images --region eu-west-1 --repository-name artem-services --query "sort_by(imageDetails,& imagePushedAt)[ * ].imageTags[ * ]" --output text']
              def proc = command.execute()
              return proc.text.readLines()
            '''
        ]
      ]
    ]
  ])
])

AWS Cli – ECR display only image tags in the repository

Example using AWS Cli only, without third-party utilities

aws ecr describe-images \
--region eu-west-1 \
--repository-name artem-services \
--query "sort_by(imageDetails,& imagePushedAt)[ * ].imageTags[ * ]" \
--output text

 

Example using "jq" utility

aws ecr describe-images \
--region eu-west-1 \
--repository-name artem-services \
| jq '.imageIds | map (.imageTag)|sort|.[]'

Jenkins – Kubeconfig

 

To store and switch between Kubernetes configurations, you can use the Kubernetes CLI plugin, to work with it, "kubectl" must be installed in the system

Install the plugin

Go to "Manage Jenkins"

 

Section "Manage Plugins"

 

Go to the "Available" tab and in the search indicate "Kubernetes CLI"

Install it.

 

Add Kubernetes config

 

Go to the "Credentials" and create the "Secret file"

 

We load a config and we specify "ID". Save.

 

Pipeline

An example of using a plugin in Pipeline:

pipeline {
  agent any
  stages {   
    stage("K8s") {
      steps {
        withKubeConfig([credentialsId: 'kubeconfig-artem-services-staging']) {
          script {
            sh "kubectl get pod"
          }
        }
      }
    }
  }
}

FIX ERROR – CentOS 7 kubectl: Invalid choice: 'eks', maybe you meant

When trying to execute any "kubectl" command on an EKS cluster, the following error occurs:

Invalid choice: 'eks', maybe you meant:

 

A possible reason for this is the old version of "kubectl" available in the repository.

 

Solution:

Install the latest version of "kubectl" using PIP3

sudo yum install -y python3 python3-pip
sudo pip3 install --upgrade --user awscli

 

To use "kubectl" installed using PIP, you need the PIP binary files directory to be in the PATH variable, to do this:

export PATH=~/.local/bin:$PATH
source ~/.bash_profile

 

In order not to do this every time, you need to add the line:

export PATH=~/.local/bin:$PATH

 

In your profile, depends on the shell:

  • Bash – .bash_profile, .profile, or .bash_login
  • Zsh – .zshrc
  • Tcsh – .tcshrc, .cshrc or .login

Jenkins – Add new path to PATH

 

To add the path to the Jenkins environment globally, without doing this every time in Pipeline, we’ll go to the Jenkins settings

 

Next is the tab "Configure System"

 

Find the "Global properties" block, check the "Environment variables" checkbox. And we add a variable with the name "PATH + EXTRA" and the value – the paths that need to be added, separated by ":" among themselves, if there are several.

Helm – Creating Secret from Variable

 

In order to save the value of the variable as Secret, the variable must be encoded in base64, for this we use "_helpers.tpl"

For example, we need to save the value of the variable "applicationSecret"

values.yaml:

# Default values for artem-services.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1
namespace: default

image:
  repository: XXXXXXXXXXXX.dkr.ecr.eu-east-1.amazonaws.com/artem-services
  tag: 1.0.1
  pullPolicy: IfNotPresent

nameOverride: ""
fullnameOverride: "artem-services"

applicationSecret: |
  var1=value1
  var2=value2
  var3=value3
  var4=value4

 

Add the following to "_helpers.tpl":

_helpers.tpl:

{{- define "applicationSecret" }}
{{- printf .Values.applicationSecret | b64enc }}
{{- end }}

 

Now in the "templates" directory, create the file "secret.yaml"

secret.yaml:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: "{{ .Chart.Name }}-application"
data:
  application.conf: {{ template "applicationSecret" . }}

Ansible – Launch Playbook with a specific version of Python

For example, there is an instance on which Python of the 2nd and 3rd versions is installed, but the default is 2nd, and in order not to change the default version and start the Playbook using Python3, you can use the following command:

python3 $(which ansible-playbook) -i localhost my-playbook.yaml

 

You can also specify the interpreter in the inventory file:

[localhost]
localhost ansible_connection=local ansible_python_interpreter=python3

 

Make sure Ansible module is installed for the correct version of Python

sudo pip3 install --user ansible