如何删除 gitlab CI 作业管道 logs/builds 和历史
How to delete gitlab CI jobs pipelines logs/builds and history
我们如何配置 gitlab 只保留最后 10 CI jobs/builds 并继续删除其余的？
例如，在 Jenkins 中，我们可以将作业配置为仅保留最后 X 个构建。
我认为 Gitlab 不支持此功能。但是您可以使用 Gitlab API 和 webhooks 自行创建此功能。
当您推送到 repo（并启动管道）时，它将触发 webhook，它可以通过 API 读取您的 CI 历史记录 => 您可以删除任何您想要的内容。
的文档
这是 job API
的文档
仅供参考，我使用类似的解决方案。我已经为每个分支部署了服务器（每个分支都有 MR）。当 MR 关闭时，它会删除已部署的服务器。很靠谱。
更新
从 Gitlab Release 12.6 开始，删除管道现在是所有者 GUI 中的一个选项：
- 在管道列表中单击要删除的管道
- 点击管道详细信息页面右上角的红色删除按钮。
从 Gitlab Release 11.6 开始，删除管道现在是一个选项，仅供维护者通过 API。
你需要：
- 一个API令牌
- 项目
id
- 您要删除的管道
pipeline_id。
项目 id: 1 和 pipeline_id: 4 的文档中使用 curl 的示例：
curl --header "PRIVATE-TOKEN: <your_access_token>" --request "DELETE" "https://gitlab.example.com/api/v4/projects/1/pipelines/46"
文档是 here
对于懒人，展开
得到你的PROJECT和TOKEN和运行直到所有的管道都被删除
for PIPELINE in $(curl --header "PRIVATE-TOKEN: $TOKEN" "https://gitlab.com/api/v4/projects/$PROJECT/jobs?per_page=100" | jq '.[].pipeline.id') ; do
echo "deleting $PIPELINE"
curl --header "PRIVATE-TOKEN: $TOKEN" --request "DELETE" "https://gitlab.com/api/v4/projects/$PROJECT/pipelines/$PIPELINE"
done
修复了懒惰的批量删除脚本，从最旧的管道中删除 X 个管道。
注：需要jq。
#!/bin/bash
set -e
TOKEN=""
PROJECT=""
# How many to delete from the oldest.
PER_PAGE=100
for PIPELINE in $(curl --header "PRIVATE-TOKEN: $TOKEN" "https://gitlab.com/api/v4/projects/$PROJECT/pipelines?per_page=$PER_PAGE&sort=asc" | jq '.[].id') ; do
echo "Deleting pipeline $PIPELINE"
curl --header "PRIVATE-TOKEN: $TOKEN" --request "DELETE" "https://gitlab.com/api/v4/projects/$PROJECT/pipelines/$PIPELINE"
done
_gitlab_pipeline_cleanup() {
echo GITLAB PIPELINE CLEANUP TOOL
echo DELETING ALL EXCEPT RUNNING AND THE MOST RECENT PIPELINE
if [ -z $GITLABURL ]; then echo GITLAB_URL:; read GITLABURL;fi
if [ -z $GITLAB_TOKEN ]; then echo TOKEN:; read GITLAB_TOKEN;fi
if [ -z $PROJECT ]; then echo PROJECT_NUM:; read PROJECT;fi
list=$(curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "$GITLABURL/api/v4/projects/$PROJECT/pipelines?per_page=100" |jq -c '.[] | select( .status != "running" )| .id ' |tail -n+2 |grep -v ^$)
echo removing from $GITLABURL Project number $PROJECT
while echo $(echo -n "$list" |wc -c ) |grep -v ^0$; do
list=$(curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "$GITLABURL/api/v4/projects/$PROJECT/pipelines?per_page=100" |jq -c '.[] | select( .status != "running" )| .id ' |tail -n+2 |grep -v ^$)
for item in $list ;do
echo -n "| -p $item |"
curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" --request "DELETE" "$GITLABURL/api/v4/projects/$PROJECT/pipelines/$item"
done
done
echo ; } ;
那你可以
for PROJECT in 12345 23476234 2138734876238746 ;do _gitlab_pipeline_cleanup ;done
作为 Mog 回答的一个小扩展，因为一方面我的 gitlab 每页最多提供 100 个条目，另一方面我想根据管道的状态删除：
#!/bin/bash
set -e
TOKEN="---your token here--"
PROJECT="PROJECT-ID"
BASEURL="https://gitlab.com/api/v4/projects" # Base Url, if you host your
# Status values: created, waiting_for_resource, preparing, pending, running, success, failed, canceled, skipped, manual, scheduled
STATUS=(failed canceled skipped) # separate status by space
# How many to delete from the oldest.
DELETE_CNT=500
# -----
CNT=0
declare -A STATUS_RESULT
for CUR_STATUS in "${STATUS[@]}";
do
TEMP_CNT=$CNT
DOLOOP=true
while [ "$DOLOOP" = true ]
do
DOLOOP=false
for PIPELINE in $(curl --header "PRIVATE-TOKEN: $TOKEN" "$BASEURL/$PROJECT/pipelines?per_page=$DELETE_CNT&sort=asc&status=$CUR_STATUS" | jq '.[].id') ; do
if [[ "$CNT" -lt "$DELETE_CNT" ]]; then
echo "#$CNT Deleting pipeline $PIPELINE with status $CUR_STATUS"
curl --header "PRIVATE-TOKEN: $TOKEN" --request "DELETE" "$BASEURL/$PROJECT/pipelines/$PIPELINE"
let "CNT+=1"
DOLOOP=true
fi
done
if [ "$DOLOOP" = true ] ; then
if [[ "$CNT" -ge "$DELETE_CNT" ]]; then
DOLOOP=false
fi
fi
done
TEMP_CNT=$((CNT-TEMP_CNT))
STATUS_RESULT[$CUR_STATUS]=$TEMP_CNT
echo "Removed $TEMP_CNT pipelines with status $CUR_STATUS"
done
echo "===================================================="
echo "= Summary of removed pipelines (max: $DELETE_CNT)"
echo "= Total: $CNT"
echo "="
for key in "${!STATUS_RESULT[@]}"; do
CNT=${STATUS_RESULT[$key]}
echo "= $key: $CNT"
done
根据之前的答案，修改脚本以检索多个项目，并为每个项目删除早于配置日期的管道。
#!/bin/bash
set -e
TOKEN=""
# How many to delete from the oldest.
PER_PAGE=100
UPDATED_BEFORE=2021-02-01T00:00:00Z
GITLAB_URL=
while : ; do
COUNTER=0
for PROJECT in $(curl -s --header "PRIVATE-TOKEN: $TOKEN" "$GITLAB_URL/api/v4/projects?per_page=$PER_PAGE" | jq '.[].id') ; do
echo "Deleting in project $PROJECT"
for PIPELINE in $(curl -s --header "PRIVATE-TOKEN: $TOKEN" "$GITLAB_URL/api/v4/projects/$PROJECT/pipelines?per_page=$PER_PAGE&sort=asc&updated_before=$UPDATED_BEFORE" | jq '.[].id') ; do
echo "Deleting pipeline $PIPELINE"
curl --header "PRIVATE-TOKEN: $TOKEN" --request "DELETE" "$GITLAB_URL/api/v4/projects/$PROJECT/pipelines/$PIPELINE"
(( COUNTER++ )) || true
done
done
echo $COUNTER
if [[ "$COUNTER" -le 0 ]]; then
break;
fi
done
这会删除项目的所有 管道，但我相信您可以找出 Perl 内容以跳过前 10
curl -s --header "PRIVATE-TOKEN: ******" --request "GET" "https://gitlab.com/api/v4/projects/********/pipelines?per_page=32767" \
| perl -MJSON -le '$d = decode_json(<>); map { print $_->{"id"} } @$d' - | while read i; do
curl -s --header "PRIVATE-TOKEN: ********" --request "DELETE" "https://gitlab.com/api/v4/projects/******/pipelines/$i"
done
项目 ID 写在项目名称（项目页面）下，您可以从“编辑配置文件”获得访问令牌。“访问令牌”并选中“API”复选框。
在 Linux 上通过以下方式在 Perl 中安装 JSON 模块：
sudo apt -y install libjson-perl
要使用 python 删除项目中的所有管道，可以使用以下代码。您可以前往 Jupyter 在线试用
import requests
def confirmDeletion():
confirmDelete = input('Do you want to delete all pipelines in the project Y/N ?')
if confirmDelete == 'Y' or confirmDelete == 'y' :
return True
else:
return False
projectId = input('Provide the Gitlab Project ID')
token = input('Provide the Gitlab Access Token')
proceed = bool()
pipelinesQueryUrl = f'https://gitlab.com/api/v4/projects/{projectId}/pipelines'
if confirmDeletion():
print('Deleting pipelines')
# The Gitlab API does
while True:
response = requests.get(pipelinesQueryUrl, headers = {"PRIVATE-TOKEN": token})
if response.ok:
pipelines = response.json()
if len(pipelines) == 0:
print('No more pipelines found, exiting')
break
else:
for pipeline in pipelines:
pipelineId = pipeline.get('id')
url = f'https://gitlab.com/api/v4/projects/{projectId}/pipelines/{pipelineId}'
response = requests.delete(url, headers = {"PRIVATE-TOKEN": token})
if response.ok:
print(f'Pipeline {pipelineId} succesfully deleted')
else:
print (f'Deleting pipeline {pipelineId} on path failed: {response.url} : ({response.status_code}) {response.reason}')
if response.status_code == 429:
# Watch out for rate specific limits https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlabcom-specific-rate-limits
print ('Rate Limits have been reached, wait and try again later')
break
else:
print (f'Querying for pipelines failed: {response.url}: ({response.status_code}) {response.reason}')
if response.status_code == 429:
# Watch out for rate specific limits https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlabcom-specific-rate-limits
print ('Rate Limits have been reached, wait and try again later')
break
else:
print('No pipelines deleted')
我们如何配置 gitlab 只保留最后 10 CI jobs/builds 并继续删除其余的？
例如，在 Jenkins 中，我们可以将作业配置为仅保留最后 X 个构建。
我认为 Gitlab 不支持此功能。但是您可以使用 Gitlab API 和 webhooks 自行创建此功能。
当您推送到 repo（并启动管道）时，它将触发 webhook，它可以通过 API 读取您的 CI 历史记录 => 您可以删除任何您想要的内容。
的文档这是 job API
的文档仅供参考，我使用类似的解决方案。我已经为每个分支部署了服务器（每个分支都有 MR）。当 MR 关闭时，它会删除已部署的服务器。很靠谱。
更新
从 Gitlab Release 12.6 开始，删除管道现在是所有者 GUI 中的一个选项：
- 在管道列表中单击要删除的管道
- 点击管道详细信息页面右上角的红色删除按钮。
从 Gitlab Release 11.6 开始，删除管道现在是一个选项，仅供维护者通过 API。
你需要：
- 一个API令牌
- 项目
id - 您要删除的管道
pipeline_id。
项目 id: 1 和 pipeline_id: 4 的文档中使用 curl 的示例：
curl --header "PRIVATE-TOKEN: <your_access_token>" --request "DELETE" "https://gitlab.example.com/api/v4/projects/1/pipelines/46"
文档是 here
对于懒人，展开
得到你的PROJECT和TOKEN和运行直到所有的管道都被删除
for PIPELINE in $(curl --header "PRIVATE-TOKEN: $TOKEN" "https://gitlab.com/api/v4/projects/$PROJECT/jobs?per_page=100" | jq '.[].pipeline.id') ; do
echo "deleting $PIPELINE"
curl --header "PRIVATE-TOKEN: $TOKEN" --request "DELETE" "https://gitlab.com/api/v4/projects/$PROJECT/pipelines/$PIPELINE"
done
修复了懒惰的批量删除脚本，从最旧的管道中删除 X 个管道。
注：需要jq。
#!/bin/bash
set -e
TOKEN=""
PROJECT=""
# How many to delete from the oldest.
PER_PAGE=100
for PIPELINE in $(curl --header "PRIVATE-TOKEN: $TOKEN" "https://gitlab.com/api/v4/projects/$PROJECT/pipelines?per_page=$PER_PAGE&sort=asc" | jq '.[].id') ; do
echo "Deleting pipeline $PIPELINE"
curl --header "PRIVATE-TOKEN: $TOKEN" --request "DELETE" "https://gitlab.com/api/v4/projects/$PROJECT/pipelines/$PIPELINE"
done
_gitlab_pipeline_cleanup() {
echo GITLAB PIPELINE CLEANUP TOOL
echo DELETING ALL EXCEPT RUNNING AND THE MOST RECENT PIPELINE
if [ -z $GITLABURL ]; then echo GITLAB_URL:; read GITLABURL;fi
if [ -z $GITLAB_TOKEN ]; then echo TOKEN:; read GITLAB_TOKEN;fi
if [ -z $PROJECT ]; then echo PROJECT_NUM:; read PROJECT;fi
list=$(curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "$GITLABURL/api/v4/projects/$PROJECT/pipelines?per_page=100" |jq -c '.[] | select( .status != "running" )| .id ' |tail -n+2 |grep -v ^$)
echo removing from $GITLABURL Project number $PROJECT
while echo $(echo -n "$list" |wc -c ) |grep -v ^0$; do
list=$(curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "$GITLABURL/api/v4/projects/$PROJECT/pipelines?per_page=100" |jq -c '.[] | select( .status != "running" )| .id ' |tail -n+2 |grep -v ^$)
for item in $list ;do
echo -n "| -p $item |"
curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" --request "DELETE" "$GITLABURL/api/v4/projects/$PROJECT/pipelines/$item"
done
done
echo ; } ;
那你可以
for PROJECT in 12345 23476234 2138734876238746 ;do _gitlab_pipeline_cleanup ;done
作为 Mog 回答的一个小扩展，因为一方面我的 gitlab 每页最多提供 100 个条目，另一方面我想根据管道的状态删除：
#!/bin/bash
set -e
TOKEN="---your token here--"
PROJECT="PROJECT-ID"
BASEURL="https://gitlab.com/api/v4/projects" # Base Url, if you host your
# Status values: created, waiting_for_resource, preparing, pending, running, success, failed, canceled, skipped, manual, scheduled
STATUS=(failed canceled skipped) # separate status by space
# How many to delete from the oldest.
DELETE_CNT=500
# -----
CNT=0
declare -A STATUS_RESULT
for CUR_STATUS in "${STATUS[@]}";
do
TEMP_CNT=$CNT
DOLOOP=true
while [ "$DOLOOP" = true ]
do
DOLOOP=false
for PIPELINE in $(curl --header "PRIVATE-TOKEN: $TOKEN" "$BASEURL/$PROJECT/pipelines?per_page=$DELETE_CNT&sort=asc&status=$CUR_STATUS" | jq '.[].id') ; do
if [[ "$CNT" -lt "$DELETE_CNT" ]]; then
echo "#$CNT Deleting pipeline $PIPELINE with status $CUR_STATUS"
curl --header "PRIVATE-TOKEN: $TOKEN" --request "DELETE" "$BASEURL/$PROJECT/pipelines/$PIPELINE"
let "CNT+=1"
DOLOOP=true
fi
done
if [ "$DOLOOP" = true ] ; then
if [[ "$CNT" -ge "$DELETE_CNT" ]]; then
DOLOOP=false
fi
fi
done
TEMP_CNT=$((CNT-TEMP_CNT))
STATUS_RESULT[$CUR_STATUS]=$TEMP_CNT
echo "Removed $TEMP_CNT pipelines with status $CUR_STATUS"
done
echo "===================================================="
echo "= Summary of removed pipelines (max: $DELETE_CNT)"
echo "= Total: $CNT"
echo "="
for key in "${!STATUS_RESULT[@]}"; do
CNT=${STATUS_RESULT[$key]}
echo "= $key: $CNT"
done
根据之前的答案，修改脚本以检索多个项目，并为每个项目删除早于配置日期的管道。
#!/bin/bash
set -e
TOKEN=""
# How many to delete from the oldest.
PER_PAGE=100
UPDATED_BEFORE=2021-02-01T00:00:00Z
GITLAB_URL=
while : ; do
COUNTER=0
for PROJECT in $(curl -s --header "PRIVATE-TOKEN: $TOKEN" "$GITLAB_URL/api/v4/projects?per_page=$PER_PAGE" | jq '.[].id') ; do
echo "Deleting in project $PROJECT"
for PIPELINE in $(curl -s --header "PRIVATE-TOKEN: $TOKEN" "$GITLAB_URL/api/v4/projects/$PROJECT/pipelines?per_page=$PER_PAGE&sort=asc&updated_before=$UPDATED_BEFORE" | jq '.[].id') ; do
echo "Deleting pipeline $PIPELINE"
curl --header "PRIVATE-TOKEN: $TOKEN" --request "DELETE" "$GITLAB_URL/api/v4/projects/$PROJECT/pipelines/$PIPELINE"
(( COUNTER++ )) || true
done
done
echo $COUNTER
if [[ "$COUNTER" -le 0 ]]; then
break;
fi
done
这会删除项目的所有 管道，但我相信您可以找出 Perl 内容以跳过前 10
curl -s --header "PRIVATE-TOKEN: ******" --request "GET" "https://gitlab.com/api/v4/projects/********/pipelines?per_page=32767" \
| perl -MJSON -le '$d = decode_json(<>); map { print $_->{"id"} } @$d' - | while read i; do
curl -s --header "PRIVATE-TOKEN: ********" --request "DELETE" "https://gitlab.com/api/v4/projects/******/pipelines/$i"
done
项目 ID 写在项目名称（项目页面）下，您可以从“编辑配置文件”获得访问令牌。“访问令牌”并选中“API”复选框。
在 Linux 上通过以下方式在 Perl 中安装 JSON 模块：
sudo apt -y install libjson-perl
要使用 python 删除项目中的所有管道，可以使用以下代码。您可以前往 Jupyter 在线试用
import requests
def confirmDeletion():
confirmDelete = input('Do you want to delete all pipelines in the project Y/N ?')
if confirmDelete == 'Y' or confirmDelete == 'y' :
return True
else:
return False
projectId = input('Provide the Gitlab Project ID')
token = input('Provide the Gitlab Access Token')
proceed = bool()
pipelinesQueryUrl = f'https://gitlab.com/api/v4/projects/{projectId}/pipelines'
if confirmDeletion():
print('Deleting pipelines')
# The Gitlab API does
while True:
response = requests.get(pipelinesQueryUrl, headers = {"PRIVATE-TOKEN": token})
if response.ok:
pipelines = response.json()
if len(pipelines) == 0:
print('No more pipelines found, exiting')
break
else:
for pipeline in pipelines:
pipelineId = pipeline.get('id')
url = f'https://gitlab.com/api/v4/projects/{projectId}/pipelines/{pipelineId}'
response = requests.delete(url, headers = {"PRIVATE-TOKEN": token})
if response.ok:
print(f'Pipeline {pipelineId} succesfully deleted')
else:
print (f'Deleting pipeline {pipelineId} on path failed: {response.url} : ({response.status_code}) {response.reason}')
if response.status_code == 429:
# Watch out for rate specific limits https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlabcom-specific-rate-limits
print ('Rate Limits have been reached, wait and try again later')
break
else:
print (f'Querying for pipelines failed: {response.url}: ({response.status_code}) {response.reason}')
if response.status_code == 429:
# Watch out for rate specific limits https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlabcom-specific-rate-limits
print ('Rate Limits have been reached, wait and try again later')
break
else:
print('No pipelines deleted')