Skip to content

Jenkins-Kubernetes自动化构建

自动化部署方案流程

mermaid
flowchart TD
    A[开发人员提交代码] --> B[GitLab 代码仓库]
    B --> C[Jenkins-Master 检测到变更]
    C --> D[Jenkins-Master 分配任务]
    D --> E[Jenkins-Agent-01 执行构建]
    E --> F{构建阶段}
    F --> G[从 Nexus 获取依赖]
    G --> H[Maven/Gradle 构建项目]
    H --> I[生成制品(JAR/WAR)]
    I --> J[Docker 构建镜像]
    J --> K[推送镜像到 Harbor]
    K --> L[Jenkins 触发部署]
    L --> M[Kubernetes 集群]
    M --> N[从 Harbor 拉取镜像]
    N --> O[部署应用 Pod]
    O --> P[服务发布]
    P --> Q[用户访问应用]
    
    style A fill:#e1f5fe
    style B fill:#f3e5f5
    style C fill:#fff3e0
    style E fill:#ffecb3
    style G fill:#bbdefb
    style I fill:#c8e6c9
    style J fill:#ffcdd2
    style K fill:#e1bee7
    style N fill:#dcedc8
    style O fill:#ffcc80
    style Q fill:#a5d6a7

流程阶段说明

  1. 代码提交与触发:开发人员将代码推送到 GitLab 仓库,Jenkins-Master 通过 Webhook 检测到代码变更。
  2. 任务分配:Jenkins-Master 将构建任务分配给 Jenkins-Agent-01 执行。
  3. 构建阶段
    • Jenkins-Agent-01 从 Nexus 私有仓库获取依赖包(如 JAR 文件),加速构建过程。
    • 使用 Maven 或 Gradle 编译、测试和打包项目,生成制品(如 JAR 或 WAR 包)。
  4. 镜像构建与推送
    • 根据项目中的 Dockerfile 构建 Docker 镜像。
    • 将构建好的镜像推送到 Harbor 私有镜像仓库进行存储和管理。
  5. 部署阶段
    • Jenkins 触发 Kubernetes 集群进行部署。
    • Kubernetes 从 Harbor 拉取镜像。
    • 部署应用 Pod 并发布服务,最终用户可以通过服务访问应用。

关键组件交互

  • Jenkins-Master:负责流程调度和任务管理。
  • Jenkins-Agent-01:实际执行构建任务,需安装 Docker 和 JDK 等工具。
  • GitLab:存储源代码,并通过 Webhook 触发 Jenkins 构建。
  • Nexus:作为 Jar 包缓存仓库,提供构建依赖。
  • Harbor:存储 Docker 镜像,确保镜像安全和管理。
  • Kubernetes:负责应用的部署和运维。

配置注意事项

  1. Jenkins 凭据配置:在 Jenkins 中正确配置访问 GitLab、Harbor 和 Nexus 的凭据(用户名/密码或 Token)。
  2. Webhook 设置:确保 GitLab 的 Webhook 指向 Jenkins-Master 的 URL,并设置好触发规则(如 Push 事件)。
  3. Jenkins Agent 连接:确保 jenkins-agent-01 节点已正确连接到 Jenkins-Master,并具有执行 Docker 命令的权限。
  4. Kubernetes 权限:配置 Jenkins 或相关 Agent 具有操作 Kubernetes API 的权限(如使用 kubeconfig 文件或 ServiceAccount)。

环境准备

HostnameIP操作系统说明
jenkins-master192.168.148.111openEuler-24.03(LTS)部署Jenkins服务
jenkins-agent-01192.168.148.112openEuler-24.03(LTS)部署Docker服务,用于执行Jenkins-agent执行job任务
gitlab192.168.148.110openEuler-24.03(LTS)部署Gitlab服务,私有代码仓库
horbor192.168.148.130openEuler-24.03(LTS)部署Horbor,私有镜像仓库
nexus192.168.148.114openEuler-24.03(LTS)部署Nexus服务,缓存jar包
k8s-master192.168.148.180openEuler-24.03(LTS)Kubernetes 集群主节点(kubeadm部署)
k8s-node01192.168.148.181openEuler-24.03(LTS)Kubernetes 集群从节点
k8s-node01192.168.148.182openEuler-24.03(LTS)Kubernetes 集群从节点

安装 Jenkins插件

安装

Manager jenkins —> Plugin —> Available plugins —> 搜索Docker,可以选择DockerDocker ComposesDocker Pipeline

配置

Manager Jenkins —> Clouds—> New cloud —> Type选择Docker

Docker Cloud details

Docker 部分,配置Docker主机的连接信息。如果Jenkins运行在Docker主机上,直接使用Unix socket/var/run/docker.sock即可。配置Docker镜像、容器和构建参数。

  • Cloud name:Docker-28.33
  • Docker Host URI:tcp:192.168.148.112:2375主机IP填写docker部署的主机IP
  • Test Connection:出现Docker版本信息链接成功Version = 28.3.3, API Version = 1.51

Docker Agent templates

创建模板,镜像最好需要自定义镜像,因为涉及到凭证问题

  • Labels:自定义如:docker-on-slave

  • Enabled:勾选是启动否则是禁用

  • Name:自定义如:inbound-agent

  • Docker Image:官网镜像jenkins/inbound-agent:latest,最好自定义镜像

  • Containers settings:容器设置要求容器内必须有Java

    • Docker Command:启动容器的指令,适用于自定义构建的镜像

    • Hostname:主机名

    • User:指定容器运行的用户

    • Extra Groups:指定容器运行的组

      挂载docker.sock涉及到权限问题,需要设置Extra Groups

      bash
      # 启动容器会指定--group-add=998
      getent group docker | cut -d: -f3
    • DNS:设置容器的DNS,如果没有设置使用主机DNS

    • DNS Search:设置容器DNS搜索域,如果没有设置使用主机配置

    • Network:设置容器网络,可以自定义如:jenkins

    • Mounts:容器挂载

      bash
      # typel:挂载类型volume|bind|tmpfs|npipe
      # [src|source]:挂载源
      # dst|destination|target:容器内部位置
      # 容器内拉取代码涉及到指纹和密钥问题,可以在宿主机上提前配置好然后挂载到容器内,最好是自定义构建的镜像比如利用jenkins/inbound-agent:latest作为基础镜像然后构建专属Node或者mvn构建的镜像
      
      type=bind,src=/data/jk,dst=/data
      type=bind,src=/home/jenkins/.ssh/,dst=/home/jenkins/.ssh,ro
      
      # 如果需要在容器在构建镜像需要挂载docker.sock
      type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock

      挂载docker.sock涉及到权限问题,需要设置Extra Groups

    • Volumes From:从其它容器挂载

    • Shared Memory Size in MB:内存可以根据服务器配置分配

  • Pull strategy:选择Never pull,自定义镜像会去Github拉取镜像

配置jenkins-agent节点

  • 安装好Docker环境后需要配置Docker的2375端口,用于jenkins-master节点方便连接配置hosts
  • insecure-registries配置Horbor私有镜像仓库地址
bash
cat > /etc/docker/daemon.json <<'EOF'
{
    "data-root":"/data/docker",
    "storage-driver":"overlay2",
    "registry-mirrors":[
        "https://docker.m.daocloud.io",
        "https://noohub.ru",
        "https://huecker.io",
        "https://dockerhub.timeweb.cloud"
    ],
    "insecure-registries": ["192.168.148.130:80"],
    "bip":"172.18.0.1/16",
    "exec-opts":["native.cgroupdriver=systemd"],
    "ipv6": false,
    "insecure-registries": ["192.168.148.130:80"],
    "hosts": ["tcp://0.0.0.0:2375", "unix:///var/run/docker.sock"]
}
EOF
systemctl daemon-reload
systemctl restart docker

配置Kubernetes

创建命名空间

bash
kubectl create ns ys

创建Harbor凭证

bash
kubectl create secret docker-registry harbor-regcred \
--docker-server=192.168.148.130:80 \
--docker-username=admin \
--docker-password=Harbor12345 \
--docker-email=email@example.com \
-n ys
  • --docker-server:你的Harbor仓库地址,必须和镜像地址中的一致
  • --docker-username:替换为你的Harbor用户名
  • --docker-password:替换为对应用户的密码
  • --docker-email:可选,但建议提供邮箱地址
  • -n:指定命名空间

配置Horbor私有镜像仓库地址

在所有k8s-node节点设置Horbor私有仓库地址

bash
cat > /etc/docker/daemon.json <<'EOF'
{
    "insecure-registries": ["192.168.148.130:80"]
}
EOF
systemctl daemon-reload
systemctl restart docker

外部数据库的 IP 映射(可选)

生产环境Kubernetes集群的pod中请求集群外部数据库地址,需要通过创建Endpoint + Service方案来解决。

创建 Endpoint

bash
cat > ys-db-endpoints.yaml << 'EOF'
apiVersion: v1
kind: Endpoints
metadata:
  name: mysql-external  # 需与 Service 同名
  namespace: ys
subsets:
  - addresses:
      - ip: 192.168.148.171  # 外部 MySQL IP
    ports:
      - port: 3306
---
apiVersion: v1
kind: Endpoints
metadata:
  name: redis-external
  namespace: ys
subsets:
  - addresses:
      - ip: 192.168.148.52  # 外部 Redis IP
    ports:
      - port: 6379
EOF
kubectl apply -f ys-db-endpoints.yaml

创建 Service

bash
cat > ys-db-service.yaml << 'EOF'
apiVersion: v1
kind: Service
metadata:
  name: mysql-external  # 与 Endpoint 同名
  namespace: ys
spec:
  ports:
    - port: 3306
  type: ClusterIP  # 仅集群内访问
---
apiVersion: v1
kind: Service
metadata:
  name: redis-external
  namespace: ys
spec:
  ports:
    - port: 6379
EOF
kubectl apply -f ys-db-service.yaml

在 Pod 中通过 Service 名称访问

bash
# MySQL 连接地址
mysql-external.default.svc.cluster.local:3306

# Redis 连接地址
redis-external.default.svc.cluster.local:6379

自动化构建前端

构建jenkins-agent-node镜像

jenkins官方提供的agent镜像jenkins/inbound-agent只有Java和Git环境,并没有构建环境,所以需要自行解决构建环境。前端需要NodeJS环境。

在jenkins-agent-01,因为Jenkins会自动连接到这个节点上的Docker执行Job任务

bash
cat > jenkins-agent-node-dockerfile << 'EOF'
FROM jenkins/inbound-agent:jdk17
ARG NODE_VERSION=v22.18.0
USER root
ADD https://nodejs.org/dist/${NODE_VERSION}/node-${NODE_VERSION}-linux-x64.tar.gz /tmp/
RUN cd  /tmp && \
	tar xf node-${NODE_VERSION}-linux-x64.tar.gz && \
	mv node-${NODE_VERSION}-linux-x64 /usr/local/nodejs && \
	chown -R jenkins:jenkins /usr/local/nodejs && \
	ln -s /usr/local/nodejs/bin/* /usr/local/bin/ && \
	rm -rf node-${NODE_VERSION}-linux-x64.tar.gz && \
	rm -rf /usr/local/nodejs/include share LICENSE README.md CHANGELOG.md && \
	npm config set registry https://registry.npmmirror.com/ --global
USER jenkins
WORDIR /data
EOF

docker build -t jenkins-agent-node:v1.0 -f jenkins-agent-node-dockerfile .

构建业务Dockerfile

自定义构建前端项目的镜像使用自定义Nginx镜像作为基础镜像,将这个Dockerfile上传到Gitlab代码仓库里,方便在后续的流水线构建。

bash
cat > Dockerfile << 'EOF'
FROM nginx-web:v1.0.0-alpine
COPY dist.tar.gz /tmp/
RUN rm -rf /data/wwwroot/www/* && \
    tar xf /tmp/dist.tar.gz && \
	mv dist/* /data/wwwroot/www
EXPOSE 80
CMD ["nginx","-g","daemon off;"]
EOF

# 提交代码
git add .
git commit -m "feat:增加Dockerfile"
git push origin main

创建Kuernetes资源文件

资源文件上传到代码仓库中方便后续的自动化构建操作

bash
cat > deploy-web.yaml << 'EOF'
---
# 无状态应用
apiVersion: apps/v1
kind: Deployment
metadata:
 name: yshopmall-web
 namespace: ys
spec:
  replicas: 2
  selector:
    matchLabels:
      app: yshopmall-web
  template:
    metadata:
      name: yshopmall-web
      namespace: ys
      labels:
        app: yshopmall-web
    spec:
      imagePullSecrets:
        - name: harbor-regcred
      containers:
        - name: yshopmall-web
          image: 192.168.148.130:80/web/yshopmall_qd:v1.0
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
          startupProbe:
            httpGet:
              path: /
              port: 80
              scheme: HTTP
            initialDelaySeconds: 3
---
# Service
apiVersion: v1
kind: Service
metadata:
  name: yshopmall-web
  namespace: ys
  labels:
    app: yshopmall-web
spec:
  type: NodePort
  selector:
    app: yshopmall-web
  ports:
    - name: yshopmall-web
      port: 80
      nodePort: 30080
      targetPort: 80
EOF
# 提交代码
git add .
git commit -m "feat:增加deploy资源"
git push origin main

Pipeline流水线

语法可以参考http://192.168.148.111:8080/job/yshopmall_qd/pipeline-syntax/

groovy
pipeline {
    agent {
        label 'jenkins-agent-node'
    }
    environment{
        APP_VERSION = 'v1.0.1'
        NODE_OPTIONS = "--openssl-legacy-provider"
        APP_NAME = 'yshopmall_qd'
        HORBOR_URL = '192.168.148.130:80'
    }
    stages {
        stage ('拉取代码'){
            steps{
                sshagent(credentials: ['gitlab-ssh-key']) {
                    git branch: 'main', url: "git@192.168.148.110:yshopmall/${env.APP_NAME}.git"
                }
                sh '''
                cat > .env.production << 'EOF'
ENV = 'production'
VUE_APP_BASE_API  = 'http://192.168.148.180:30081'
VUE_APP_WS_API = 'ws://192.168.148.180:30081'
'''
                sh 'npm install'
                sh 'npm run build'
                sh 'tar zcf dist.tar.gz dist'
            }
        }
        stage('构建WEB') {
            steps {
                script {
                    dockerImage = docker.build("${env.APP_NAME}:${env.APP_VERSION}")
                }
            }
        }
        stage('上传镜像到harbor') {
            steps {
                sh """
                echo 'Harbor12345' | docker login -u admin http://${env.HORBOR_URL} --password-stdin
                """
                sh "docker tag ${env.APP_NAME}:${env.APP_VERSION} ${env.HORBOR_URL}/web/${env.APP_NAME}:${env.APP_VERSION}"
                sh "docker push ${env.HORBOR_URL}/web/${env.APP_NAME}:${env.APP_VERSION}"
                sh "docker rmi ${env.APP_NAME}:${env.APP_VERSION}"
                sh "docker rmi ${env.HORBOR_URL}/web/${env.APP_NAME}:${env.APP_VERSION}"
            }
        }
        stage('部署远程资源'){
            steps {
                sshPublisher(
                    publishers: [
                        sshPublisherDesc(
                            configName: 'k8s-master',
                            sshCredentials: [
                                encryptedPassphrase: '{AQAAABAAAAAQKePJwv21CBk0wAQwlu3s4Vysb8mrfKXRw01HIxrq3wA=}',
                                username: 'root'
                            ],
                            transfers: [
                                sshTransfer(
                                    sourceFiles: 'deploy-web.yaml',
                                    remoteDirectory: '/deploy/',
                                    removePrefix: '',
                                    execCommand: """
                                        kubectl apply -f /data/deploy/deploy-web.yaml || 
                                        echo 'Deployment failed!'
                                    """,
                                    execTimeout: 120000,
                                    usePty: true
                                )
                            ],
                            verbose: true
                        )
                    ]
                )
            }
        }
        stage('清理工作空间') {
            steps {
                script {
                    // 删除整个工作目录
                    deleteDir()
                }
            }
        }
    }
}

自动化构建后端

构建jenkind-agent-maven镜像

settings-nexus(可选)

bash
cat > settings.xml << 'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.2.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.2.0 https://maven.apache.org/xsd/settings-1.2.0.xsd">
  <pluginGroups></pluginGroups>
  <proxies></proxies>
  <servers>
    <!-- nexus -->
    <server>
      <id>my-nexus-releases</id>
      <username>admin</username>
        <password>admin123</password>
    </server>
    <!-- nexus -->
    <server>
      <id>my-nexus-snapshots</id>
       <username>admin</username>
        <password>admin123</password>
    </server>
  </servers>
  <mirrors>
    <mirror>
      <id>maven-default-http-blocker</id>
      <mirrorOf>external:http:*</mirrorOf>
      <name>Pseudo repository to mirror external repositories initially using HTTP.</name>
      <url>http://0.0.0.0/</url>
      <blocked>true</blocked>
    </mirror>
    <!-- nexus -->
    <mirror>
      <id>nexus</id>
      <mirrorOf>*</mirrorOf>
      <url>http://192.168.148.114:8081/repository/maven-public/</url>
    </mirror>
  </mirrors>
  <profiles>
    <!-- nexus -->
    <profile>
      <id>nexus</id>
      <repositories>
        <repository>
          <id>nexus</id>
          <url>http://192.168.148.114:8081/repository/maven-public/</url>
          <releases><enabled>true</enabled></releases>
          <snapshots> <enabled>true</enabled></snapshots>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
          <id>nexus</id>
          <url>http://192.168.148.114:8081/repository/maven-public/</url>
          <releases><enabled>true</enabled></releases>
          <snapshots><enabled>true</enabled></snapshots>
        </pluginRepository>
      </pluginRepositories>
    </profile>
  </profiles>
  <!-- nexus -->
  <activeProfiles>
    <activeProfile>nexus</activeProfile>
  </activeProfiles>
</settings>
EOF

settings-aliyu(可选)

bash
cat > settings.xml << 'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.2.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.2.0 https://maven.apache.org/xsd/settings-1.2.0.xsd">
  <pluginGroups></pluginGroups>
  <proxies></proxies>
  <servers></servers>
  <mirrors>
    <mirror>
      <id>maven-default-http-blocker</id>
      <mirrorOf>external:http:*</mirrorOf>
      <name>Pseudo repository to mirror external repositories initially using HTTP.</name>
      <url>http://0.0.0.0/</url>
      <blocked>true</blocked>
    </mirror>
    <!-- alimaven -->
    <mirror>
        <id>alimaven</id>
        <mirrorOf>*</mirrorOf>
        <name>Aliyun Global Maven</name>
        <url>https://maven.aliyun.com/repository/public/</url>
    </mirror>
  </mirrors>
  <profiles>
  </profiles>
</settings>
EOF

Dockerfile

bash
cat > jenkins-agent-maven-dockerfile << 'EOF'
FROM jenkins/inbound-agent:jdk17
ARG MAVEN_VERSION=3.8.9
USER root
COPY settings.xml /tmp/
ADD https://mirrors.tuna.tsinghua.edu.cn/apache/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz /tmp/
RUN cd /tmp && \
	tar xf apache-maven-${MAVEN_VERSION}-bin.tar.gz -C /usr/local && \
	mv -f settings.xml /usr/local/apache-maven-${MAVEN_VERSION}/conf/ && \
	chown -R jenkins:jenkins /usr/local/apache-maven-${MAVEN_VERSION} && \
	ln -s /usr/local/apache-maven-${MAVEN_VERSION}/bin/* /usr/local/bin/ && \
	rm -rf apache-maven-${MAVEN_VERSION}-bin.tar.gz && \
	mkdir -p /home/jenkins/.m2  && \
    chown -R jenkins:jenkins  /home/jenkins/.m2
USER jenkins
WORKDIR /app
EOF

docker build -t jenkins-agent-maven:v1.0 -f jenkins-agent-maven-dockerfile .

构建业务Dockerfile

自定义构建前端项目的镜像使用自定义Nginx镜像作为基础镜像,将这个Dockerfile上传到Gitlab代码仓库里,方便在后续的流水线构建。

bash
cat > Dockerfile << 'EOF'
FROM openjdk:8-jdk-alpine
ENV TZ=Asia/Shanghai
ENV JAVA_OPTS="-Xms1024m -Xmx2048m -Djava.security.egd=file:/dev/./urandom -Djava.awt.headless=true"

RUN ln -sf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone && \
	mkdir -p /yshop-admin && \
  	sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories && \
  	apk add --no-cache fontconfig ttf-dejavu

WORKDIR /yshop-admin
EXPOSE 8001
ADD ./target/yshop-admin-3.3.jar ./
CMD ["sh", "-c", "java $JAVA_OPTS -jar yshop-admin-3.3.jar --spring.profiles.active=prod"]

EOF

# 提交代码
git add .
git commit -m "feat:增加Dockerfile"
git push origin main

创建Kuernetes资源文件

资源文件上传到代码仓库中方便后续的自动化构建操作

bash
cat > deploy-java.yaml << 'EOF'
---
# 无状态应用
apiVersion: apps/v1
kind: Deployment
metadata:
 name: yshopmall-java
 namespace: ys
spec:
  replicas: 2
  selector:
    matchLabels:
      app: yshopmall-java
  template:
    metadata:
      name: yshopmall-java
      namespace: ys
      labels:
        app: yshopmall-java
    spec:
      imagePullSecrets:
        - name: harbor-regcred
      containers:
        - name: yshopmall-java
          image: 192.168.148.130:80/java/yshopmall:v1.0.1
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 8001
              scheme: HTTP
            initialDelaySeconds: 3
            failureThreshold: 5
          resources:
            limits:
              cpu: 200m
              memory: 1024Mi
            requests:
              cpu: 200m
              memory: 1024Mi

---
# Service
apiVersion: v1
kind: Service
metadata:
  name: yshopmall-java
  namespace: ys
  labels:
    app: yshopmall-java
spec:
  type: NodePort
  selector:
    app: yshopmall-java
  ports:
    - name: yshopmall-java
      port: 8001
      nodePort: 30081
      targetPort: 8001
EOF
# 提交代码
git add .
git commit -m "feat:增加deploy资源"
git push origin main

Pipeline流水线

groovy
pipeline {
    agent {
        label 'jenkins-agent-maven'
    }
    environment{
        APP_VERSION = 'v1.0.1'
        APP_NAME = 'yshopmall'
        HORBOR_URL = '192.168.148.130:80'
    }
    stages {
        stage ('拉取代码'){
            steps{
                sshagent(credentials: ['gitlab-ssh-key']) {
                    git branch: 'main', url: "git@192.168.148.110:yshopmall/${env.APP_NAME}.git"
                }
                sh "sed -i 's/localhost/192.168.148.171/' yshop-admin/src/main/resources/config/application-docker.yml"
                sh "sed -i 's/3366/3306/' yshop-admin/src/main/resources/config/application-docker.yml"
                sh "sed -i 's/username: yshopb2c/username: root/' yshop-admin/src/main/resources/config/application-docker.yml"
                sh "sed -i 's/bkfGfAimifjPZtNE/123456/' yshop-admin/src/main/resources/config/application-docker.yml"
                sh "sed -i 's/host: 127.0.0.1/host: 192.168.148.52/' yshop-admin/src/main/resources/config/application-docker.yml"
                sh "sed -i 's/port: 6399/port: 6379/' yshop-admin/src/main/resources/config/application-docker.yml"
                sh "sed -i 's/password: 6379@@6379/password: 123456/' yshop-admin/src/main/resources/config/application-docker.yml"
                sh 'mvn clean package'
            }
        }
        stage('Build镜像') {
            steps {
                script {
                    dir('yshop-admin') {
                        dockerImage = docker.build("${env.APP_NAME}:${env.APP_VERSION}", ".")
                    }
                }
            }
        }
        stage('上传镜像到harbor') {
            steps {
                sh """
                echo 'Harbor12345' | docker login -u admin http://${env.HORBOR_URL} --password-stdin
                """
                sh "docker tag ${env.APP_NAME}:${env.APP_VERSION} ${env.HORBOR_URL}/java/${env.APP_NAME}:${env.APP_VERSION}"
                sh "docker push ${env.HORBOR_URL}/java/${env.APP_NAME}:${env.APP_VERSION}"
                sh "docker rmi ${env.APP_NAME}:${env.APP_VERSION}"
                sh "docker rmi ${env.HORBOR_URL}/java/${env.APP_NAME}:${env.APP_VERSION}"
            }
        }
        stage('远程启动资源'){
            steps {
                sshPublisher(
                    publishers: [
                        sshPublisherDesc(
                            configName: 'k8s-master',
                            sshCredentials: [
                                encryptedPassphrase: '{AQAAABAAAAAQKePJwv21CBk0wAQwlu3s4Vysb8mrfKXRw01HIxrq3wA=}',
                                username: 'root'
                            ],
                            transfers: [
                                sshTransfer(
                                    sourceFiles: 'deploy-java.yaml',
                                    remoteDirectory: '/deploy/',
                                    removePrefix: '',
                                    execCommand: """
                                        kubectl apply -f /data/deploy/deploy-java.yaml || 
                                        echo 'Deployment failed!'
                                    """,
                                    execTimeout: 120000,
                                    usePty: true
                                )
                            ],
                            verbose: true
                        )
                    ]
                )
            }
        }
        // stage('清理工作空间') {
        //     steps {
        //         script {
        //             // 删除整个工作目录
        //             deleteDir()
        //         }
        //     }
        // }
    }
}