k8s jenkins master-slave setting

System Management–System Configuration–Node Management–Configure Clouds–Configure Cluster

Kubernetes

Name: kubernetes-prod
Kubernetes address: https://kubernetes.default.svc.cluster.local
Use Jenkins Proxy: unchecked
Kubernetes service certificate key: empty
Disable HTTPS certificate checking: Unchecked
Kubernetes namespace: empty
Credentials: None
WebSocket: Unchecked
Direct Connection: unchecked
Jenkins address: http://jenkins.dayi-devops.svc.cluster.local:8080/jenkins
Jenkins channel: empty
Connection Timeout: 30
Read Timeout: 60
Number of Containers: 20
Pod Labels:
1. Pod Labels:
  1. key: jenkins
  2. Value: jnlp
2. Maximum number of connections to the Kubernetes API: 32
3. Seconds to wait for pods to be running: 600
Pod Templates
1. Pod Templates
  1. Name: jnlp-slave
  2. Namespace: prod
  3. List of tags: jnlp-slave
  4. Usage: only allow jobs bound to this machine to run
  5. Parent’s Pod Template Name: empty
container list
1. Container Template
  1. Name: jnlp
  2. Docker image: wanyan.cn-hangzhou.cr.aliyuncs.com/yyh-prod/jenkins-jnlp:v2
  3. Always pull images: unchecked
  4. Working directory: /home/jenkins
  5. Command to run: jenkins-agent
  6. Command parameters: empty
  7. Assign Pseudo-Terminal: Checked
2. Environment Variables: (the default configuration does not operate)
3. environment variable
  1. Volume (Host Path Volume)
    1. Host path: /var/run/docker.sock
    2. Mount path: /var/run/docker.sock
  2. Host Path Volume
    1. Host path: /usr/bin/docker
    2. Mount path: /usr/bin/docker
  3. Host Path Volume
    1. Host path: /etc/localtime
    2. Mount path: /etc/localtime
  4. Persistent Volume Claim
    1. Declaration value: webapps-data
    2. Read-only: unchecked
    3. Mount path: /data/webapps
  5. Notes
    1. Concurrency Limit: empty
    2. Pod Retention: Default
    3. Agent’s idle survival time (minutes): empty
    4. Pod Lifetime (seconds): Empty
    5. Timeout (seconds) to connect to Jenkins: 1000
    6. Raw YAML for the Pod: empty
    7. Yaml merge strategy: Override
    8. Show raw yaml in console: check
4. Pull the Secret of the mirror
  1. Image Pull Secret
    1. Name: aliregistry-secret
    2. Service Account: jenkins-prod
    3. Run As User ID: 0 (started by root user)
    4. Run As Group ID: empty
    5. Supplemental Groups: empty
    6. Host Network: Unchecked
    7. node selector: empty
  2. Workspace Volume: Persistent Volume Claim Workspace Volume
    1. Declaration value: jenkins-jnlp-local
    2. Read-only: unchecked
  3. node properties
    1. Tool location: unchecked

Jenkins front-end application configuration

Parametric build process
1. Git parameters
  1. Name: Branch
  2. Description: Select the branch for the release
  3. Argument type: branch or label
  4. Default: master
2. option parameter
  1. Name: Namespace
  2. option: prod
  3. Description: Select a publishing environment
3. option parameter
  1. Name: deploy_env
  2. option: deploy
  3. Description: deploy releases new code

Pipeline script

script

// item
// Need to modify the directory where the front-end project is deployed
def project_webdir = "channelcenter"
// Need to modify the git address of the corresponding service
def git_address = "http://gitlab.wanyan.com/web/channel-center-web.git"
// authentication
def git_auth = "gitlab-creds" //git login auth


pipeline {
    agent { label 'jnlp-slave' }

    parameters {
        gitParameter branch: '', branchFilter: '.*', defaultValue: 'master', description: 'Select branch for release', name: 'Branch', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH_TAG' //PT_BRANCH_TAG gets branch and TAG
        //gitParameter branch: '', branchFilter: '.*', defaultValue: 'master', description: 'select the branch to publish', name: 'Branch', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH' //PT_BRANCH only gets branches
        choice (choices: ['prod'], description: 'select the publishing environment', name: 'Namespace')
        choice choices: ['deploy'], description: '''deploy release new code''', name: 'deploy_env'
    }
    
    stages {
        stage('Pull code'){
            steps {
                //build quietPeriod: 3, job: 'yyh_devops'
                checkout([$class: 'GitSCM',
                branches: [[name: "${params.Branch}"]],
                doGenerateSubmoduleConfigurations: false,
                extensions: [], submoduleCfg: [],
                userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]
                ])
            }
        }

        stage('code compilation'){
           when { environment name: 'deploy_env', value: 'deploy' }
           steps {
             sh """
                cnpm install --unsafe-perm --registry=https://registry.npm.taobao.org
                cnpm run build:prod
                pwd
                """
           }
        }

        stage('deployment') {
           when { environment name: 'deploy_env', value: 'deploy' }
           steps {
             sh """
                rsync -avz --delete dist/ /data/webapps/${project_webdir}
                pwd
                """
           }
        }

    }
}

Jenkins backend application configuration

Parametric build process
1. Git parameters
  1. Name: Branch
  2. Description: Select the branch for the release
  3. Argument type: branch or label
  4. Default: master
2. option parameter
  1. Name: Namespace
  2. option: prod
  3. Description: Select a publishing environment
3. option parameter
  1. Name: deploy_env
  2. Options: deploy or rollback
  3. Description: deploy releases new code rollback rollback
4. character parameter
  1. Name: version
  2. Default: 0
  3. Description: Select rollback version number
  4. Clear blank characters: Unchecked

Pipeline script

pipeline {
    agent { label 'jnlp-slave' }

    environment {
        // public
        registry = "wanyan.cn-hangzhou.cr.aliyuncs.com"
        // project
        project = "yyh-prod"
        app_name = "${JOB_NAME}"
        workdir = "/home/dayiops/${JOB_NAME}"
        image_name = "${registry}/${project}/${app_name}:${BUILD_NUMBER}"
        // Need to modify the port number of the corresponding service
        app_port = "8890"
        // Need to modify the git address of the corresponding service
        git_address = "http://gitlab.wanyan.com/basic-service/auth-center.git"
        // Need to modify the gitlab group corresponding to the microservice
        git_groups = "basic-service"
        // Roll back the mirrored version
        rollback_image_name = "${registry}/${project}/${app_name}:${version}"
        // authentication
        docker_registry_auth = "jenkins-aliregistry-creds" //Harbor login auth
        git_auth = "gitlab-creds" //git login auth

    }

    parameters {
        gitParameter branch: '', branchFilter: '.*', defaultValue: 'master', description: 'Select branch for release', name: 'Branch', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH_TAG' //PT_BRANCH_TAG gets branch and TAG
        //gitParameter branch: '', branchFilter: '.*', defaultValue: 'master', description: 'select the branch to publish', name: 'Branch', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH' //PT_BRANCH only gets branches
        //choice (choices: ['1', '3', '5', '7'], description: 'Replica Count', name: 'ReplicaCount')
        choice (choices: ['prod'], description: 'select the publishing environment', name: 'Namespace')
        choice choices: ['deploy', 'rollback'], description: '''deploy release new code rollback rollback''', name: 'deploy_env'
        string defaultValue: '0', description: 'Select rollback version number', name: 'version', trim: false
    }
    
    stages {

            stage('Pull code'){
                steps {
                    build quietPeriod: 3, job: 'yyh_devops'
                    checkout([$class: 'GitSCM',
                    branches: [[name: "${params.Branch}"]],
                    doGenerateSubmoduleConfigurations: false,
                    extensions: [], submoduleCfg: [],
                    userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]
                    ])
                }
            }
    
            stage('code compilation'){
                when { environment name: 'deploy_env', value: 'deploy' }
                steps {
                  sh """
                     mvn clean package -Dmaven.test.skip=true -U
                     """
                }
            }
    
            stage('build image'){
                when { environment name: 'deploy_env', value: 'deploy' }
                steps {
                    dir("${WORKSPACE}/${JOB_NAME}") {
                     withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
                     sh """
                       echo '
                         FROM ${registry}/yyh/centos-jdk:8-jre
                         LABEL author='yhh'
                         
                         ENV PROJECT="${JOB_NAME}"
                         
                         #ENV JAVA_OPTS="-Dspring.profiles.active=production-server -Xms1024M -Xmx1024M"
                         ENV TZ=Asia/Shanghai
                         ENV LANG=en_US.UTF-8
     
                         RUN mkdir /home/dayiops/${JOB_NAME} -p
                         
                         WORKDIR /home/dayiops/${JOB_NAME}
                         
                         ADD target/${JOB_NAME}.jar ${workdir}/
                         
                         EXPOSE ${app_port}/tcp
                         
                         ENTRYPOINT ["java","-Dspring.profiles.active=k8s${Namespace}","-Dmaven.wagon.http.ssl.insecure=true","-Dmaven.wagon.http .ssl.allowall=true","-server","-Xms1024M","-Xmx1024M","-XX: + HeapDumpOnOutOfMemoryError","-jar","$ {JOB_NAME}.jar"]
                       '>Dockerfile
                       docker login -u ${username} -p '${password}' ${registry}
                       docker build -t ${image_name} .
                       docker push ${image_name}
                     """
                     }
                    }
                }
            }
    
            stage('deploy to K8S platform'){
                when { environment name: 'deploy_env', value: 'deploy' }
                steps {
                  dir("$WORKSPACE/../yyh_devops/${git_groups}/${JOB_NAME}") {
                     sh """
                       sed -i 's#{APP_NAME}#${JOB_NAME}#g' k8s-deployment.yaml
                       sed -i 's#{APP_PORT}#${app_port}#g' k8s-deployment.yaml
                       sed -i 's#{IMAGE_NAME}#${image_name}#' k8s-deployment.yaml
                       sed -i 's#{NAME_SPACE}#${Namespace}#' k8s-deployment.yaml
                       sed -i 's#{ADD_ENV_LABEL}#${Namespace}#' k8s-deployment.yaml
                       kubectl apply -f k8s-deployment.yaml
                     """
                     //kubernetesDeploy configs: 'k8s-deployment.yaml'
                   }
                }
            }
  
            stage("Service startup check"){
                when { environment name: 'deploy_env', value: 'deploy' }
                steps {
                    sleep 63
                    timeout(time: 31, unit: 'SECONDS') {
                        waitUntil {
                            script {
                                def podstatus = sh (
                                    returnStdout: true,
                                    //script: "kubectl get deployment -n test | grep auth-center-api | awk \'{print \$1}\'"
                                    script: "kubectl get replicasets -n ${Namespace} |grep ${JOB_NAME} | awk \'{if (\$2 >=1 & amp; & amp; \$4 == 0) print \ "podnotready"}\'"
                                )
                                def notrun_podname = sh (
                                    returnStdout: true,
                                    //script: "kubectl get deployment -n test | grep ${JOB_NAME} | awk \'{print \$1}\'"
                                    script: "kubectl get pod -n ${Namespace} |grep ${JOB_NAME} |awk \'{if (\$2 == "0/1") print \$1} \'"
                                )
                                podstatus = podstatus. trim()
                                notrun_podname = notrun_podname.trim()
                                echo "********${JOB_NAME} service startup status is ${podstatus}********"
                                if( podstatus == "podnotready" ) {
                                    //echo "${JOB_NAME} service failed to start, rechecking service running status..."
                                    echo "Service startup status checking..."
                                    sleep 10
                                    return false
                                } else {
                                    echo "********${JOB_NAME} service started successfully...********"
                                    return true
                                }
                            }
                       }
                    }
                }
            }
    
            stage('roll back the specified image'){
                when { environment name: 'deploy_env', value: 'rollback' }
                steps {
                    dir("$WORKSPACE/../yyh_devops/${git_groups}/${JOB_NAME}") {
                      sh """
                      sed -i 's#{APP_NAME}#${JOB_NAME}#g' k8s-deployment.yaml
                      sed -i 's#{APP_PORT}#${app_port}#g' k8s-deployment.yaml
                      sed -i 's#{IMAGE_NAME}#${rollback_image_name}#' k8s-deployment.yaml
                      sed -i 's#{NAME_SPACE}#${Namespace}#' k8s-deployment.yaml
                      sed -i 's#{ADD_ENV_LABEL}#${Namespace}#' k8s-deployment.yaml
                      kubectl apply -f k8s-deployment.yaml
                      """
                      //kubernetesDeploy configs: 'k8s-deployment.yaml'
                    }
                }
            }
    }
}

Backend application Dockerfile template

# cat Dockerfile

                     FROM dayi-registry.cn-hangzhou.cr.aliyuncs.com/yyh/centos-jdk:8-jre
                     LABEL author=yhh

                     ENV PROJECT="gateway"

                     #ENV JAVA_OPTS="-Dspring.profiles.active=production-server -Xms1024M -Xmx1024M"
                     ENV TZ=Asia/Shanghai

                     RUN mkdir /home/dayiops/gateway -p

                     WORKDIR /home/dayiops/gateway

                     ADD target/gateway.jar /home/dayiops/gateway/

                     EXPOSE 8081/tcp

                     ENTRYPOINT ["java","-Dspring.profiles.active=k8sprod","-server","-Xms2048M","-Xmx2048M","-XX: + HeapDumpOnOutOfMemoryError\ ","-jar","gateway.jar"]

gitlab k8s template

#kubernetes/yyh-devops/BC/connector-api/k8s-deployment.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: {APP_NAME}
  namespace: {NAME_SPACE}
  labels:
    app: {APP_NAME}
    env: {ADD_ENV_LABEL}
spec:
  ports:
  - name: http
    port: {APP_PORT}
    protocol: TCP
    targetPort: {APP_PORT}
  selector:
    app: {APP_NAME}
    env: {ADD_ENV_LABEL}
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {APP_NAME}
  namespace: {NAME_SPACE}
  labels:
    app: {APP_NAME}
    env: {ADD_ENV_LABEL}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: {APP_NAME}
      env: {ADD_ENV_LABEL}
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: {APP_NAME}
        env: {ADD_ENV_LABEL}
    spec:
      imagePullSecrets:
      - name: aliregistry-secret
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - {APP_NAME}
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      -env:
        - name: TZ
          value: Asia/Shanghai
        -name: LANG
          value: en_US.UTF-8
        image: {IMAGE_NAME}
        imagePullPolicy: IfNotPresent
        name: {APP_NAME}
        ports:
        - name: http
          containerPort: {APP_PORT}
          protocol: TCP
        readinessProbe:
          failureThreshold: 2
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: {APP_PORT}
          timeoutSeconds: 2
        livenessProbe:
          failureThreshold: 2
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: {APP_PORT}
          timeoutSeconds: 2
        resources:
          limits:
            cpu: 1000m
            memory: 1024Mi
          requests:
            cpu: 200m
            memory: 256Mi
        volumeMounts:
        - mountPath: /data/logs
          name: logs
        - mountPath: /etc/localtime
          name: localtime
          readOnly: true
      dnsPolicy: ClusterFirstWithHostNet
      restartPolicy: Always
      securityContext:
        fsGroup: 2049
        runAsGroup: 2049
        runAsUser: 2049
      volumes:
      - emptyDir: {}
        name: logs
      - hostPath:
          path: /etc/localtime
          type: File
        name: localtime

Dependent task configuration (this task will be executed every time the backend is released)

Name: yyh_devops

Pipeline yyh_devops

Pipeline script

node('jnlp-slave') {
    stage('Git Clone') {
        git credentialsId: 'gitlab-creds', url: 'http://gitlab.taeteadata.com/kubernetes/yyh-devops-prod.git'
    }
}