Jenkins 计划的备份管道 - 确保所有作业在 运行 之前停止
Jenkins scheduled backup pipeline - Make sure all jobs are stopped before run
我有一个 Jenkins 管道 运行 整夜进行备份。
我需要找到一种方法，仅在所有其他作业完成后才 运行 管道，并确保同时不会触发其他作业。
这应该类似于这个插件 https://plugins.jenkins.io/exclusive-execution/ 正在做的事情。
但是声明性管道不支持“独占执行”。
我附上了我的 Jenkinsfile，以防有人需要查看：
def BACKUP_DATETIME = new Date().format('yyyy_MM_dd_HH_mm_ss', TimeZone.getTimeZone('Canada/Eastern'))
pipeline {
agent { label 'master' }
options {
//Build options
disableResume()
disableConcurrentBuilds()
buildDiscarder(
logRotator (
artifactDaysToKeepStr: '10',
artifactNumToKeepStr: '1',
daysToKeepStr: '30',
numToKeepStr: '30'
)
)
}
triggers { cron('TZ=Canada/Eastern\n0 3 * * *') }
environment {
GIT_SSH_KEY_ID = 'gitlab_builduser'
BACKUP_PREFIX_NAME = 'jenkins-back-'
BACKUP_REPO_NAME = "jenkins_backups"
BACKUP_REPO_URL = "git@XXX:XX/${env.BACKUP_REPO_NAME}.git"
NUMBER_OF_BACKUPS = '7'
GOOGLE_CHAT_TOKEN = 'XXXXXF'
JENKINS_VERSION = ''
}
stages {
stage('Init') {
steps {
script {
env.JENKINS_VERSION = sh(script: "java -jar /usr/share/jenkins/jenkins.war --version", returnStdout: true).trim()
}
dir("/tmp") {
withCredentials([sshUserPrivateKey(credentialsId: "${env.GIT_SSH_KEY_ID}", keyFileVariable: 'SSH_KEY', usernameVariable: 'SSH_USER')]) {
withEnv(["GIT_SSH_COMMAND=ssh -i ${SSH_KEY}"]) {
sh '''
if [ ! -d ${BACKUP_REPO_NAME} ]
then
git clone ${BACKUP_REPO_URL}
else
cd ${BACKUP_REPO_NAME}
git fetch origin
git reset --hard origin/master
fi
'''
}
}
}
}
}
stage('Backup') {
steps {
dir ('jenkins_backup') {
sh """
chmod +x ./jenkins-backup.sh
./jenkins-backup.sh ${JENKINS_HOME} /tmp/${env.BACKUP_REPO_NAME}/${BACKUP_PREFIX_NAME}_${BACKUP_DATETIME}.tar.gz
"""
}
}
}
stage('Upload') {
steps {
// TO_DO: SAVE THE BACKUP file
dir("/tmp/${env.BACKUP_REPO_NAME}") {
withCredentials([sshUserPrivateKey(credentialsId: "${env.GIT_SSH_KEY_ID}", keyFileVariable: 'SSH_KEY', usernameVariable: 'SSH_USER')]) {
withEnv(["GIT_SSH_COMMAND=ssh -i ${SSH_KEY}"]) {
sh '''
git config user.email "${SSH_USER}@email.com"
git config user.name "${SSH_USER}"
# backup only keep the lastest X backup-files
ls -tp ${BACKUP_PREFIX_NAME}_*.tar.gz | grep -v '/$' | tail -n +$(expr ${NUMBER_OF_BACKUPS} + 1) | xargs -I {} rm -- {}
# Push to git
git add --all
git status
git commit --amend -CHEAD --allow-empty
git push --force-with-lease origin master
env
'''
}
}
}
}
}
}
post {
failure {
hangoutsNotify message: "${JOB_BASE_NAME} faild!",token: "${env.GOOGLE_CHAT_TOKEN}",threadByJob: true
}
}
}
如果您在一个节点上 运行 这些作业，有一种简单的方法可以实现此目的。使用 BuildBLocker 插件。
从这里安装：
https://plugins.jenkins.io/build-blocker-plugin/
示例用法如下所示：
pipelineJob('MyPipeline') {
// Block build if certain jobs are running.
blockOn('.*AnotherPipeline.*') {
// Possible values are 'GLOBAL' and 'NODE' (default).
blockLevel('GLOBAL')
// Possible values are 'ALL', 'BUILDABLE' and 'DISABLED' (default).
scanQueueFor('DISABLED')
}
}
您也可以直接从 GUI 执行此操作：
并设置附加选项：
希望对您有所帮助。问候！
我最终编写了一个 groovy 脚本来满足我的需要。该脚本将列出除备份作业之外的所有当前 运行 和排队的作业，如果其中一个列表不为空，则放置一个计时器。当所有作业都处于空闲状态时，备份将继续 运行。
我会在这里附上我的脚本：
詹金斯文件：
def BACKUP_DATETIME = new Date().format('yyyy_MM_dd_HH_mm_ss', TimeZone.getTimeZone('Canada/Eastern'))
pipeline {
agent { label 'master' }
options {
//Build options
disableResume()
disableConcurrentBuilds()
buildDiscarder(
logRotator (
artifactDaysToKeepStr: '10',
artifactNumToKeepStr: '1',
daysToKeepStr: '30',
numToKeepStr: '30'
)
)
}
triggers { cron('TZ=Canada/Eastern\n0 3 * * *') }
environment {
GIT_SSH_KEY_ID = 'gitlab_builduser'
BACKUP_PREFIX_NAME = 'jenkins-backup'
BACKUP_REPO_NAME = "jenkins_backups"
BACKUP_REPO_URL = "git@XXXX/${env.BACKUP_REPO_NAME}.git"
NUMBER_OF_BACKUPS = '7'
GOOGLE_CHAT_TOKEN = 'XXXX'
JENKINS_VERSION = ''
}
stages {
stage('Init') {
steps {
script {
env.JENKINS_VERSION = sh(script: "java -jar /usr/share/jenkins/jenkins.war --version", returnStdout: true).trim()
def safeBackupModule = load("${WORKSPACE}/jenkins_backup/safe_backup_run.groovy")
safeBackupModule.jenkins_safe_backup("${env.JOB_NAME}")
}
dir("/tmp") {
withCredentials([sshUserPrivateKey(credentialsId: "${env.GIT_SSH_KEY_ID}", keyFileVariable: 'SSH_KEY', usernameVariable: 'SSH_USER')]) {
withEnv(["GIT_SSH_COMMAND=ssh -i ${SSH_KEY}"]) {
sh '''
if [ ! -d ${BACKUP_REPO_NAME} ]
then
git clone ${BACKUP_REPO_URL}
else
cd ${BACKUP_REPO_NAME}
git fetch origin
git reset --hard origin/master
fi
'''
}
}
}
}
}
stage('Backup') {
steps {
dir ('jenkins_backup') {
sh """
chmod +x ./jenkins-backup.sh
./jenkins-backup.sh ${JENKINS_HOME} /tmp/${env.BACKUP_REPO_NAME}/${BACKUP_PREFIX_NAME}_${BACKUP_DATETIME}.tar.gz
"""
}
}
}
stage('Upload') {
steps {
// TO_DO: SAVE THE BACKUP file
dir("/tmp/${env.BACKUP_REPO_NAME}") {
withCredentials([sshUserPrivateKey(credentialsId: "${env.GIT_SSH_KEY_ID}", keyFileVariable: 'SSH_KEY', usernameVariable: 'SSH_USER')]) {
withEnv(["GIT_SSH_COMMAND=ssh -i ${SSH_KEY}"]) {
sh '''
git config user.email "${SSH_USER}@XXX.com"
git config user.name "${SSH_USER}"
# backup only keep the lastest X backup-files
ls -tp ${BACKUP_PREFIX_NAME}_*.tar.gz | grep -v '/$' | tail -n +$(expr ${NUMBER_OF_BACKUPS} + 1) | xargs -I {} rm -- {}
# Push to git
git add --all
git status
git commit --amend -CHEAD --allow-empty
git push --force-with-lease origin master
env
'''
}
}
}
}
}
}
post {
failure {
hangoutsNotify message: "${JOB_BASE_NAME} faild!",token: "${env.GOOGLE_CHAT_TOKEN}",threadByJob: true
}
}
}
Groovy 脚本“safe_backup_run.groovy”:
import hudson.model.RestartListener
import jenkins.model.*;
//backup method
def jenkins_safe_backup(job_name) {
//set the
if(!binding.hasVariable('timeout_seconds')) {
timeout_seconds = 60
}
if(timeout_seconds in String) {
timeout_seconds = Integer.decode(timeout_seconds)
}
//type check user defined parameters/bindings
if(!(timeout_seconds in Integer)) {
throw new Exception('PARAMETER ERROR: timeout_seconds must be an integer.')
}
print("Jenkins safe backup initiated.")
while(true) {
//get the list of all the currenty running jobs
runningBuilds = Jenkins.instance.getView('All').getBuilds().findAll() { it.getResult().equals(null) }
//remove the 'Jenkins_Backup' job from the list
runningBuilds.removeAll { it.toString().toLowerCase().startsWith(job_name.toLowerCase()) }
//get the list of queued jobs
def queuedBuilds = []
Jenkins.instance.queue.items.findAll { queuedBuilds.add(it.task.name) }
//remove the 'Jenkins_Backup' job from the list
queuedBuilds.removeAll { it.toString().toLowerCase().startsWith(job_name.toLowerCase()) }
//If no job is running except 'Jenkins_Backup' break
if((runningBuilds.isEmpty()) && (queuedBuilds.isEmpty())) {
break
}
print("Jenkins jobs/queue are not idle. Waiting ${timeout_seconds} seconds before next backup attempt.")
print("RUNNING jobs: " + runningBuilds)
print("QUEUED jobs: " + queuedBuilds)
sleep(timeout_seconds)
}
}
return this
我有一个 Jenkins 管道 运行 整夜进行备份。 我需要找到一种方法，仅在所有其他作业完成后才 运行 管道，并确保同时不会触发其他作业。 这应该类似于这个插件 https://plugins.jenkins.io/exclusive-execution/ 正在做的事情。 但是声明性管道不支持“独占执行”。
我附上了我的 Jenkinsfile，以防有人需要查看：
def BACKUP_DATETIME = new Date().format('yyyy_MM_dd_HH_mm_ss', TimeZone.getTimeZone('Canada/Eastern'))
pipeline {
agent { label 'master' }
options {
//Build options
disableResume()
disableConcurrentBuilds()
buildDiscarder(
logRotator (
artifactDaysToKeepStr: '10',
artifactNumToKeepStr: '1',
daysToKeepStr: '30',
numToKeepStr: '30'
)
)
}
triggers { cron('TZ=Canada/Eastern\n0 3 * * *') }
environment {
GIT_SSH_KEY_ID = 'gitlab_builduser'
BACKUP_PREFIX_NAME = 'jenkins-back-'
BACKUP_REPO_NAME = "jenkins_backups"
BACKUP_REPO_URL = "git@XXX:XX/${env.BACKUP_REPO_NAME}.git"
NUMBER_OF_BACKUPS = '7'
GOOGLE_CHAT_TOKEN = 'XXXXXF'
JENKINS_VERSION = ''
}
stages {
stage('Init') {
steps {
script {
env.JENKINS_VERSION = sh(script: "java -jar /usr/share/jenkins/jenkins.war --version", returnStdout: true).trim()
}
dir("/tmp") {
withCredentials([sshUserPrivateKey(credentialsId: "${env.GIT_SSH_KEY_ID}", keyFileVariable: 'SSH_KEY', usernameVariable: 'SSH_USER')]) {
withEnv(["GIT_SSH_COMMAND=ssh -i ${SSH_KEY}"]) {
sh '''
if [ ! -d ${BACKUP_REPO_NAME} ]
then
git clone ${BACKUP_REPO_URL}
else
cd ${BACKUP_REPO_NAME}
git fetch origin
git reset --hard origin/master
fi
'''
}
}
}
}
}
stage('Backup') {
steps {
dir ('jenkins_backup') {
sh """
chmod +x ./jenkins-backup.sh
./jenkins-backup.sh ${JENKINS_HOME} /tmp/${env.BACKUP_REPO_NAME}/${BACKUP_PREFIX_NAME}_${BACKUP_DATETIME}.tar.gz
"""
}
}
}
stage('Upload') {
steps {
// TO_DO: SAVE THE BACKUP file
dir("/tmp/${env.BACKUP_REPO_NAME}") {
withCredentials([sshUserPrivateKey(credentialsId: "${env.GIT_SSH_KEY_ID}", keyFileVariable: 'SSH_KEY', usernameVariable: 'SSH_USER')]) {
withEnv(["GIT_SSH_COMMAND=ssh -i ${SSH_KEY}"]) {
sh '''
git config user.email "${SSH_USER}@email.com"
git config user.name "${SSH_USER}"
# backup only keep the lastest X backup-files
ls -tp ${BACKUP_PREFIX_NAME}_*.tar.gz | grep -v '/$' | tail -n +$(expr ${NUMBER_OF_BACKUPS} + 1) | xargs -I {} rm -- {}
# Push to git
git add --all
git status
git commit --amend -CHEAD --allow-empty
git push --force-with-lease origin master
env
'''
}
}
}
}
}
}
post {
failure {
hangoutsNotify message: "${JOB_BASE_NAME} faild!",token: "${env.GOOGLE_CHAT_TOKEN}",threadByJob: true
}
}
}
如果您在一个节点上 运行 这些作业，有一种简单的方法可以实现此目的。使用 BuildBLocker 插件。
从这里安装： https://plugins.jenkins.io/build-blocker-plugin/
示例用法如下所示：
pipelineJob('MyPipeline') {
// Block build if certain jobs are running.
blockOn('.*AnotherPipeline.*') {
// Possible values are 'GLOBAL' and 'NODE' (default).
blockLevel('GLOBAL')
// Possible values are 'ALL', 'BUILDABLE' and 'DISABLED' (default).
scanQueueFor('DISABLED')
}
}
您也可以直接从 GUI 执行此操作：
并设置附加选项：
希望对您有所帮助。问候！
我最终编写了一个 groovy 脚本来满足我的需要。该脚本将列出除备份作业之外的所有当前 运行 和排队的作业，如果其中一个列表不为空，则放置一个计时器。当所有作业都处于空闲状态时，备份将继续 运行。
我会在这里附上我的脚本：
詹金斯文件：
def BACKUP_DATETIME = new Date().format('yyyy_MM_dd_HH_mm_ss', TimeZone.getTimeZone('Canada/Eastern'))
pipeline {
agent { label 'master' }
options {
//Build options
disableResume()
disableConcurrentBuilds()
buildDiscarder(
logRotator (
artifactDaysToKeepStr: '10',
artifactNumToKeepStr: '1',
daysToKeepStr: '30',
numToKeepStr: '30'
)
)
}
triggers { cron('TZ=Canada/Eastern\n0 3 * * *') }
environment {
GIT_SSH_KEY_ID = 'gitlab_builduser'
BACKUP_PREFIX_NAME = 'jenkins-backup'
BACKUP_REPO_NAME = "jenkins_backups"
BACKUP_REPO_URL = "git@XXXX/${env.BACKUP_REPO_NAME}.git"
NUMBER_OF_BACKUPS = '7'
GOOGLE_CHAT_TOKEN = 'XXXX'
JENKINS_VERSION = ''
}
stages {
stage('Init') {
steps {
script {
env.JENKINS_VERSION = sh(script: "java -jar /usr/share/jenkins/jenkins.war --version", returnStdout: true).trim()
def safeBackupModule = load("${WORKSPACE}/jenkins_backup/safe_backup_run.groovy")
safeBackupModule.jenkins_safe_backup("${env.JOB_NAME}")
}
dir("/tmp") {
withCredentials([sshUserPrivateKey(credentialsId: "${env.GIT_SSH_KEY_ID}", keyFileVariable: 'SSH_KEY', usernameVariable: 'SSH_USER')]) {
withEnv(["GIT_SSH_COMMAND=ssh -i ${SSH_KEY}"]) {
sh '''
if [ ! -d ${BACKUP_REPO_NAME} ]
then
git clone ${BACKUP_REPO_URL}
else
cd ${BACKUP_REPO_NAME}
git fetch origin
git reset --hard origin/master
fi
'''
}
}
}
}
}
stage('Backup') {
steps {
dir ('jenkins_backup') {
sh """
chmod +x ./jenkins-backup.sh
./jenkins-backup.sh ${JENKINS_HOME} /tmp/${env.BACKUP_REPO_NAME}/${BACKUP_PREFIX_NAME}_${BACKUP_DATETIME}.tar.gz
"""
}
}
}
stage('Upload') {
steps {
// TO_DO: SAVE THE BACKUP file
dir("/tmp/${env.BACKUP_REPO_NAME}") {
withCredentials([sshUserPrivateKey(credentialsId: "${env.GIT_SSH_KEY_ID}", keyFileVariable: 'SSH_KEY', usernameVariable: 'SSH_USER')]) {
withEnv(["GIT_SSH_COMMAND=ssh -i ${SSH_KEY}"]) {
sh '''
git config user.email "${SSH_USER}@XXX.com"
git config user.name "${SSH_USER}"
# backup only keep the lastest X backup-files
ls -tp ${BACKUP_PREFIX_NAME}_*.tar.gz | grep -v '/$' | tail -n +$(expr ${NUMBER_OF_BACKUPS} + 1) | xargs -I {} rm -- {}
# Push to git
git add --all
git status
git commit --amend -CHEAD --allow-empty
git push --force-with-lease origin master
env
'''
}
}
}
}
}
}
post {
failure {
hangoutsNotify message: "${JOB_BASE_NAME} faild!",token: "${env.GOOGLE_CHAT_TOKEN}",threadByJob: true
}
}
}
Groovy 脚本“safe_backup_run.groovy”:
import hudson.model.RestartListener
import jenkins.model.*;
//backup method
def jenkins_safe_backup(job_name) {
//set the
if(!binding.hasVariable('timeout_seconds')) {
timeout_seconds = 60
}
if(timeout_seconds in String) {
timeout_seconds = Integer.decode(timeout_seconds)
}
//type check user defined parameters/bindings
if(!(timeout_seconds in Integer)) {
throw new Exception('PARAMETER ERROR: timeout_seconds must be an integer.')
}
print("Jenkins safe backup initiated.")
while(true) {
//get the list of all the currenty running jobs
runningBuilds = Jenkins.instance.getView('All').getBuilds().findAll() { it.getResult().equals(null) }
//remove the 'Jenkins_Backup' job from the list
runningBuilds.removeAll { it.toString().toLowerCase().startsWith(job_name.toLowerCase()) }
//get the list of queued jobs
def queuedBuilds = []
Jenkins.instance.queue.items.findAll { queuedBuilds.add(it.task.name) }
//remove the 'Jenkins_Backup' job from the list
queuedBuilds.removeAll { it.toString().toLowerCase().startsWith(job_name.toLowerCase()) }
//If no job is running except 'Jenkins_Backup' break
if((runningBuilds.isEmpty()) && (queuedBuilds.isEmpty())) {
break
}
print("Jenkins jobs/queue are not idle. Waiting ${timeout_seconds} seconds before next backup attempt.")
print("RUNNING jobs: " + runningBuilds)
print("QUEUED jobs: " + queuedBuilds)
sleep(timeout_seconds)
}
}
return this