Jenkins 作业控制台输出转到/由 Splunk 监控 - 如何使用 CURL / Rest API 获取最后 N 分钟/小时/天等的数据
Jenkins jobs console output going to / monitored by Splunk - How to fetch that data using CURL / Rest API for last N minutes / hours / days etc
Splunk 版本
6.5.3.1
Splunk 构建
bf0ff7c2ab8b
Jenkins 版本 1.642.3 或 2.32.3
在每个 Jenkins master 上，都有一个 splunk 进程 运行ning。
$ ps -eAf|grep splunk
splunk 58877 1 20 Feb16 ? 42-23:27:37 splunkd -p 8089 restart
splunk 58878 58877 0 Feb16 ? 00:00:00 [splunkd pid=58877] splunkd -p 8089 restart [process-runner]
asangal 91197 91175 0 12:38 pts/2 00:00:00 grep --color=auto splunk
Splunk 进程监控/扫描 log 文件以查找我们实例中的任何 Jenkins 作业，即在 $JENKINS_HOME/jobs/<JOB_NAME>/builds/<BUILD_NUMBER>/log 文件下。
$ pwd
/opt/splunkforwarder/etc/system/local
$ cat inputs.conf
[default]
host = jenkins-master-project-prod-1-609
[monitor:///var/log/jenkins]
index = some-jenkins-prod-index-in-splunk
disabled = False
recursive = True
[monitor:///home/jenkins/jobs/.../builds/.../log]
index = some-jenkins-prod-index-in-splunk
disabled = False
recursive = True
crcSalt = <SOURCE>
...
.....
... more config code here ...
....
..
在 Splunk GUI 中，当我 运行 一个简单的查询来查找 Splunk 捕获的相同 index 并且来自任何 source （文件）的任何内容时，我确实看到有效的输出。 注意： 实际行输出是 t运行 满足的。正如您在条形图上看到的那样，数据在那里并且填充了 table。
在我的 Jenkins 工作中，我有时会收到一些警告、信息、错误（为此我已经在 Jenkins 级别使用 Log Parser Plugin），我正在尝试编写一个脚本来获取此日志Splunk 的 Jenkins 作业在过去 15、30 分钟或过去 1-7 小时或 1-30 天内的输出，并找出在给定时间段内发现了多少警告、错误等（基于某些关键字、正则表达式）。注意：有很多这样的 Jenkins masters，其中 Splunk 是 运行nings，我的目标是与 Splunk 对话并获取我需要的数据（而不是与 500 个 Jenkins masters 对话）。
我尝试了以下 CURL 命令，这些命令 returns 给我一个搜索 ID，但它没有做任何事情。
在下面的 CURL 命令中，我传递了一个更精确的查询来获取数据。我说的是在过去 30 分钟内获取 Splunk 拥有的所有信息（可以根据 GUI 添加字段），其中 index 是 some-jenkins-prod-index-in-splunk 并且日志的 source 是：/home/jenkins/jobs/*/builds/*/log（第一个 * 用于工作名称，第二个 * 用于内部版本号）然后我说在 splunk 中搜索日志，其中日志包含 lines/keywords/regex 之一（如下面使用 OR 条件列出）并以 JSON 格式显示输出。
➜ ~ p=$(cat ~/AKS/rOnly/od.p.txt)
➜ ~ curl --connect-time 10 --max-time 900 -ks https://splunk.server.mycompany.com:8089/services/search -umy_splunk_user:$p --data search='search earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on")' -d output_mode=json
{"messages":[{"type":"ERROR","text":"Method Not Allowed"}]}% ➜ ~
如您所见，它显示方法不允许。
当我在 URL 部分使用 /jobs 给出以下查询时，我得到了一个有效的搜索 ID。
➜ ~ curl --connect-time 10 --max-time 900 -ks https://splunk.server.mycompany.com:8089/services/search/jobs -umy_splunk_user:$p --data search='search earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on")' -d output_mode=json
{"sid":"1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A"}%
使用此搜索 ID，我试图访问主要日志，但它不起作用。我正在使用 jq 过滤 JSON 输出以显示在漂亮的布局中。
➜ ~ curl --connect-time 10 --max-time 900 -ks https://splunk.server.mycompany.com:8089/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A -umy_splunk_user:$p --data search='search earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on")' -d output_mode=json|jq .
{
"links": {},
"origin": "http://splunk.server.mycompany.com/services/search/jobs",
"updated": "2017-09-15T09:44:33-07:00",
"generator": {
"build": "bf0ff7c2ab8b",
"version": "6.5.3.1"
},
"entry": [
{
"name": "search earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log (\"WARNING: \" OR \"npm WARN retry\" OR \"svn: E200033: \" OR \": binary operator expected\" OR \": too many arguments\" OR \": No such file or directory\" OR \"rsync: failed to set times on\") | regex source=\".*/[0-9][0-9]*/log\" | table host, source, _raw",
"id": "http://splunk.server.mycompany.com/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A",
"updated": "2017-09-15T09:44:33.942-07:00",
"links": {
"alternate": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A",
"search.log": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/search.log",
"events": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/events",
"results": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/results",
"results_preview": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/results_preview",
"timeline": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/timeline",
"summary": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/summary",
"control": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/control"
},
"published": "2017-09-15T09:43:59.000-07:00",
"author": "my_splunk_user",
"content": {
"bundleVersion": "17557160226808436058",
"canSummarize": false,
"cursorTime": "1969-12-31T16:00:00.000-08:00",
"defaultSaveTTL": "2592000",
"defaultTTL": "600",
"delegate": "",
"diskUsage": 561152,
"dispatchState": "DONE",
"doneProgress": 1,
"dropCount": 0,
"earliestTime": "2017-09-15T09:13:58.000-07:00",
"eventAvailableCount": 0,
"eventCount": 30,
"eventFieldCount": 0,
"eventIsStreaming": true,
"eventIsTruncated": true,
"eventSearch": "search (earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log (\"WARNING: \" OR \"npm WARN retry\" OR \"svn: E200033: \" OR \": binary operator expected\" OR \": too many arguments\" OR \": No such file or directory\" OR \"rsync: failed to set times on\")) | regex source=\".*/[0-9][0-9]*/log\" ",
"eventSorting": "none",
"isBatchModeSearch": true,
"isDone": true,
"isEventsPreviewEnabled": false,
"isFailed": false,
"isFinalized": false,
"isPaused": false,
"isPreviewEnabled": false,
"isRealTimeSearch": false,
"isRemoteTimeline": false,
"isSaved": false,
"isSavedSearch": false,
"isTimeCursored": true,
"isZombie": false,
"keywords": "\"*: binary operator expected*\" \"*: no such file or directory*\" \"*: too many arguments*\" \"*npm warn retry*\" \"*rsync: failed to set times on*\" \"*svn: e200033: *\" \"*warning: *\" earliest::-30m index::some-jenkins-prod-index-in-splunk source::/home/jenkins/jobs/*/builds/*/log",
"label": "",
"latestTime": "2017-09-15T09:43:59.561-07:00",
"normalizedSearch": "litsearch ( index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log ( \"WARNING: \" OR \"npm WARN retry\" OR \"svn: E200033: \" OR \": binary operator expected\" OR \": too many arguments\" OR \": No such file or directory\" OR \"rsync: failed to set times on\" ) _time>=1505492038.000 ) | regex source=\".*/[0-9][0-9]*/log\" | fields keepcolorder=t \"_raw\" \"host\" \"source\"",
"numPreviews": 0,
"optimizedSearch": "| search (earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log (\"WARNING: \" OR \"npm WARN retry\" OR \"svn: E200033: \" OR \": binary operator expected\" OR \": too many arguments\" OR \": No such file or directory\" OR \"rsync: failed to set times on\")) | regex source=\".*/[0-9][0-9]*/log\" | table host, source, _raw",
"pid": "2174",
"priority": 5,
"remoteSearch": "litsearch ( index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log ( \"WARNING: \" OR \"npm WARN retry\" OR \"svn: E200033: \" OR \": binary operator expected\" OR \": too many arguments\" OR \": No such file or directory\" OR \"rsync: failed to set times on\" ) _time>=1505492038.000 ) | regex source=\".*/[0-9][0-9]*/log\" | fields keepcolorder=t \"_raw\" \"host\" \"source\"",
"reportSearch": "table host, source, _raw",
"resultCount": 30,
"resultIsStreaming": false,
"resultPreviewCount": 30,
"runDuration": 0.579,
"sampleRatio": "1",
"sampleSeed": "0",
"scanCount": 301,
"searchCanBeEventType": false,
"searchEarliestTime": 1505492038,
"searchLatestTime": 1505493839.21872,
"searchTotalBucketsCount": 37,
"searchTotalEliminatedBucketsCount": 0,
"sid": "1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A",
"statusBuckets": 0,
"ttl": 600,
"performance": {
"command.fields": {
"duration_secs": 0.035,
"invocations": 48,
"input_count": 30,
"output_count": 30
},
"command.regex": {
"duration_secs": 0.048,
"invocations": 48,
"input_count": 30,
"output_count": 30
},
"command.search": {
"duration_secs": 1.05,
"invocations": 48,
"input_count": 0,
"output_count": 30
},
"command.search.calcfields": {
"duration_secs": 0.013,
"invocations": 16,
"input_count": 301,
"output_count": 301
},
"dispatch.optimize.reparse": {
"duration_secs": 0.001,
"invocations": 1
},
"dispatch.optimize.toJson": {
"duration_secs": 0.001,
"invocations": 1
},
"dispatch.optimize.toSpl": {
"duration_secs": 0.001,
"invocations": 1
},
"dispatch.parserThread": {
"duration_secs": 0.048,
"invocations": 48
},
"dispatch.reduce": {
"duration_secs": 0.001,
"invocations": 1
},
"dispatch.stream.remote": {
"duration_secs": 1.05,
"invocations": 48,
"input_count": 0,
"output_count": 332320
},
"dispatch.stream.remote.mr11p01if-ztbv02090901.mr.if.mycompany.com-8081": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr11p01if-ztbv02090901.mr.if.mycompany.com-8082": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr11p01if-ztbv11204201.mr.if.mycompany.com-8081": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr11p01if-ztbv11204201.mr.if.mycompany.com-8082": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr11p01if-ztbv11204401.mr.if.mycompany.com-8081": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr11p01if-ztbv11204401.mr.if.mycompany.com-8082": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr11p01if-ztbv16142101.mr.if.mycompany.com-8081": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr11p01if-ztbv16142101.mr.if.mycompany.com-8082": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr11p01if-ztbv16142301.mr.if.mycompany.com-8081": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr11p01if-ztbv16142301.mr.if.mycompany.com-8082": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr21p01if-ztbv14080101.mr.if.mycompany.com-8081": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr21p01if-ztbv14080101.mr.if.mycompany.com-8082": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr22p01if-ztbv07132101.mr.if.mycompany.com-8081": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr22p01if-ztbv07132101.mr.if.mycompany.com-8082": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr22p01if-ztbv09013201.mr.if.mycompany.com-8081": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr22p01if-ztbv09013201.mr.if.mycompany.com-8082": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr90p01if-ztep02103701.mr.if.mycompany.com-8081": {
"duration_secs": 0.058,
"invocations": 2,
"input_count": 0,
"output_count": 16948
},
"dispatch.stream.remote.mr90p01if-ztep02103701.mr.if.mycompany.com-8082": {
"duration_secs": 0.066,
"invocations": 2,
"input_count": 0,
"output_count": 14415
},
"dispatch.stream.remote.mr90p01if-ztep04044101.mr.if.mycompany.com-8081": {
"duration_secs": 0.059,
"invocations": 2,
"input_count": 0,
"output_count": 15858
},
"dispatch.stream.remote.mr90p01if-ztep04044101.mr.if.mycompany.com-8082": {
"duration_secs": 0.065,
"invocations": 2,
"input_count": 0,
"output_count": 11867
},
"dispatch.stream.remote.mr90p01if-ztep06024101.mr.if.mycompany.com-8081": {
"duration_secs": 0.061,
"invocations": 2,
"input_count": 0,
"output_count": 20695
},
"dispatch.stream.remote.mr90p01if-ztep06024101.mr.if.mycompany.com-8082": {
"duration_secs": 0.06,
"invocations": 2,
"input_count": 0,
"output_count": 15193
},
"dispatch.stream.remote.mr90p01if-ztep12023601.mr.if.mycompany.com-8081": {
"duration_secs": 0.063,
"invocations": 2,
"input_count": 0,
"output_count": 15932
},
"dispatch.stream.remote.mr90p01if-ztep12023601.mr.if.mycompany.com-8082": {
"duration_secs": 0.064,
"invocations": 2,
"input_count": 0,
"output_count": 14415
},
"dispatch.stream.remote.mr90p01if-ztep12043901.mr.if.mycompany.com-8081": {
"duration_secs": 0.061,
"invocations": 2,
"input_count": 0,
"output_count": 15418
},
"dispatch.stream.remote.mr90p01if-ztep12043901.mr.if.mycompany.com-8082": {
"duration_secs": 0.058,
"invocations": 2,
"input_count": 0,
"output_count": 11866
},
"dispatch.stream.remote.pv31p01if-ztbv08050801.pv.if.mycompany.com-8081": {
"duration_secs": 0.075,
"invocations": 2,
"input_count": 0,
"output_count": 15661
},
"dispatch.stream.remote.pv31p01if-ztbv08050801.pv.if.mycompany.com-8082": {
"duration_secs": 0.071,
"invocations": 2,
"input_count": 0,
"output_count": 15845
},
"dispatch.stream.remote.pv31p01if-ztbv08051001.pv.if.mycompany.com-8081": {
"duration_secs": 0.066,
"invocations": 2,
"input_count": 0,
"output_count": 14406
},
"dispatch.stream.remote.pv31p01if-ztbv08051001.pv.if.mycompany.com-8082": {
"duration_secs": 0.072,
"invocations": 2,
"input_count": 0,
"output_count": 15524
},
"dispatch.stream.remote.pv31p01if-ztbv08051201.pv.if.mycompany.com-8081": {
"duration_secs": 0.067,
"invocations": 2,
"input_count": 0,
"output_count": 16009
},
"dispatch.stream.remote.pv31p01if-ztbv08051201.pv.if.mycompany.com-8082": {
"duration_secs": 0.068,
"invocations": 2,
"input_count": 0,
"output_count": 15516
},
"dispatch.writeStatus": {
"duration_secs": 0.012,
"invocations": 7
},
"startup.configuration": {
"duration_secs": 2.045,
"invocations": 33
},
"startup.handoff": {
"duration_secs": 14.595,
"invocations": 33
}
},
"messages": [
{
"type": "INFO",
"text": "Your timerange was substituted based on your search string"
},
{
"type": "WARN",
"text": "Unable to distribute to peer named pv31p01if-ztbv08050601.pv.if.mycompany.com:8081 at uri=pv31p01if-ztbv08050601.pv.if.mycompany.com:8081 using the uri-scheme=http because peer has status=\"Down\". Please verify uri-scheme, connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information."
},
{
"type": "WARN",
"text": "Unable to distribute to peer named pv31p01if-ztbv08050601.pv.if.mycompany.com:8082 at uri=pv31p01if-ztbv08050601.pv.if.mycompany.com:8082 using the uri-scheme=http because peer has status=\"Down\". Please verify uri-scheme, connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information."
}
],
"request": {
"search": "search earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log (\"WARNING: \" OR \"npm WARN retry\" OR \"svn: E200033: \" OR \": binary operator expected\" OR \": too many arguments\" OR \": No such file or directory\" OR \"rsync: failed to set times on\") | regex source=\".*/[0-9][0-9]*/log\" | table host, source, _raw"
},
"runtime": {
"auto_cancel": "0",
"auto_pause": "0"
},
"searchProviders": [
"mr11p01if-ztbv02090901.mr.if.mycompany.com-8081",
"mr11p01if-ztbv16142101.mr.if.mycompany.com-8082",
"mr11p01if-ztbv16142301.mr.if.mycompany.com-8081",
"mr11p01if-ztbv16142301.mr.if.mycompany.com-8082",
"mr21p01if-ztbv14080101.mr.if.mycompany.com-8081",
"mr21p01if-ztbv14080101.mr.if.mycompany.com-8082",
"mr22p01if-ztbv07132101.mr.if.mycompany.com-8081",
"mr22p01if-ztbv07132101.mr.if.mycompany.com-8082",
"mr22p01if-ztbv09013201.mr.if.mycompany.com-8081",
"mr22p01if-ztbv09013201.mr.if.mycompany.com-8082",
"mr90p01if-ztep02103701.mr.if.mycompany.com-8081",
"mr90p01if-ztep02103701.mr.if.mycompany.com-8082",
"mr90p01if-ztep04044101.mr.if.mycompany.com-8081",
"mr90p01if-ztep04044101.mr.if.mycompany.com-8082",
"mr90p01if-ztep06024101.mr.if.mycompany.com-8081",
"mr90p01if-ztep06024101.mr.if.mycompany.com-8082",
"mr90p01if-ztep12023601.mr.if.mycompany.com-8081",
"mr90p01if-ztep12023601.mr.if.mycompany.com-8082",
"mr90p01if-ztep12043901.mr.if.mycompany.com-8081",
"mr90p01if-ztep12043901.mr.if.mycompany.com-8082",
"pv31p01if-ztbv08050801.pv.if.mycompany.com-8081",
"pv31p01if-ztbv08050801.pv.if.mycompany.com-8082",
"pv31p01if-ztbv08051001.pv.if.mycompany.com-8081",
"pv31p01if-ztbv08051001.pv.if.mycompany.com-8082",
"pv31p01if-ztbv08051201.pv.if.mycompany.com-8081",
"pv31p01if-ztbv08051201.pv.if.mycompany.com-8082"
]
},
"acl": {
"perms": {
"read": [
"my_splunk_user"
],
"write": [
"my_splunk_user"
]
},
"owner": "my_splunk_user",
"modifiable": true,
"sharing": "global",
"app": "search",
"can_write": true,
"ttl": "600"
}
}
],
"paging": {
"total": 1,
"perPage": 0,
"offset": 0
}
}
➜ ~
➜ ~
但是，如您所见，生成的 JSON 输出没有用，因为它没有显示或包含我可以使用的任何 Jenkins 作业的输出。
如果在 CURL 命令中，对于 Splunk URL，如果我尝试以下任何 URLs 端点，它会给我一个错误。
"search.log": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/search.log",
"events": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/events",
"results": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/results",
"results_preview": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/results_preview",
"timeline": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/timeline",
"summary": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/summary",
"control": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/control"
例如： 如果我尝试 URL.../<SEARCH_ID>/events 或 URL/.../<SEARCH_ID>/results 等，我会收到以下错误。
curl --connect-time 10 --max-time 900 -ks https://splunk.server.mycompany.com:8089/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/events -umy_splunk_user:$p --data search='search earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on")' -d output_mode=json|jq .
{
"messages": [
{
"type": "FATAL",
"text": "Method Not Allowed"
}
]
}
我正在尝试查找主机名、来源（Jenkins 作业日志的路径）、实际作业的控制台输出（我可以读取和解析以生成有意义的信息）关于最近 N 个时间段，有多少，出现错误、警告、奇怪的线条，并且取决于某些阈值，如果数字超过这些阈值，那么我需要发送电子邮件通知。
我可以编写所有这些代码，但我没有得到这里的第一个难题，即让 Splunk 吐出 Jenkins 作业的控制台输出，其中 splunk 在文件系统上监视。
最终目标是将有意义的数据以 JSON 或 CSV 格式转储到文本文件中，并将该数据转换为一些有意义的 bar/pie 图表等。
例如：如果 data.csv 包含：
age,population
<5,2704659
5-13,4499890
14-17,2159981
18-24,3853788
25-44,14106543
45-64,8819342
65-85,312463
≥85,81312463
然后使用以下文件，我可以将此原始数据转换为饼图，如下图所示。
<!DOCTYPE html>
<meta charset="utf-8">
<style>
.arc text {
font: 10px sans-serif;
text-anchor: middle;
}
.arc path {
stroke: #fff;
}
</style>
<svg width="960" height="500"></svg>
<script src="https://d3js.org/d3.v4.min.js"></script>
<script>
var svg = d3.select("svg"),
width = +svg.attr("width"),
height = +svg.attr("height"),
radius = Math.min(width, height) / 2,
g = svg.append("g").attr("transform", "translate(" + width / 2 + "," + height / 2 + ")");
var color = d3.scaleOrdinal(["#98abc5", "#8a89a6", "#7b6888", "#6b486b", "#a05d56", "#d0743c", "#ff8c00"]);
var pie = d3.pie()
.sort(null)
.value(function(d) { return d.population; });
var path = d3.arc()
.outerRadius(radius - 10)
.innerRadius(0);
var label = d3.arc()
.outerRadius(radius - 40)
.innerRadius(radius - 40);
d3.csv("data.csv", function(d) {
d.population = +d.population;
return d;
}, function(error, data) {
if (error) throw error;
var arc = g.selectAll(".arc")
.data(pie(data))
.enter().append("g")
.attr("class", "arc");
arc.append("path")
.attr("d", path)
.attr("fill", function(d) { return color(d.data.age); });
arc.append("text")
.attr("transform", function(d) { return "translate(" + label.centroid(d) + ")"; })
.attr("dy", "0.35em")
.text(function(d) { return d.data.age; });
});
</script>
生成的饼图（由于csv文件和html文件）：
找到解决方案。
我只需要使用 services/search/jobs/export 终点。
让我们找出什么是我们的 Jenkins host（其中包含 Jenkins 作业），作业的名称是什么（这可以从日志文件的 source 路径中解析/搜索）以及 Jenkins 作业的实际控制台输出是什么 (_raw)。此外，让我们将搜索限制在最近 30 分钟内（即 earliest=-30m）。
实际上有3种方法可以做到。
1) 通过在命令行传递用户名和密码。
2) 通过生成一个 SESSION TOKEN，我们可以在 header 后的任何未来 CURL 命令中使用它。
3) 通过生成 --cookie "${COOKIE}" ID 并使用它。这是所有方法中的首选方法，因为它将 cookie 值复制到 Splunk 使用的任何后端服务器。要使用的 Cookie 名称：splunkd_8081
后两种解决方案取决于第一种方法，即使用用户凭据创建 SESSION 或 COOKIE ID。
解决方案一：
1) 这里我们将使用我们的 splunk 服务器
2) 在命令行中传递用户名和密码
3) 提供用于查找/获取 Splunk 数据的 Splunk 选项（对于包含特定行的 Jenkins 日志），并进行一些额外的正则表达式匹配（这样它将 return 与源的 Jenkins build# 完全一致路径而不是为相同的控制台输出显示 source 的 3 行。Jenkins latestBuild、latestSuccessfulBuild 等是指向编号构建的符号链接，我们不想在我们的列表中列出这些符号链接的 source 条目输出，所以我使用正则表达式模式来查找 source 路径，其中它在源路径中的 log 文件之前包含一个 NUMBERED build#。
4) 然后，我使用 | 仅过滤掉 3 个字段：host、source 和 _raw（Splunk 会 return). host 包含，哪个 Jenkins 服务器有 Jenkins 作业。 source 的值中包含 Jenkins 作业名称、build# 等信息。 _raw 变量包含 Jenkins 作业的控制台输出（靠近我们试图在 Jenkins 作业的控制台输出中搜索的字符串/行的几行）。
注意： 所有这 3 个字段都在字典变量中可用 result 所以我只是输出它。
5) 然后，我以 json 格式提供输出（您也可以使用 csv）。最后我使用 jq 来过滤信息。
注意： 如果您使用 jq -r ".result._raw"（即字典变量 result 中的 _raw 字段，那么它将逐行显示控制台输出的输出（而不是给你一个嵌入了 \n 的 blob）。你也可以使用 sed 's/\n/\n/g' 但是 jq -r ".result._raw" 很简单够了）。
命令 运行:
$ p="$(cat ~/my_secret_password.txt)"
$
$ # The above command will set my password in variable 'p'
$
$ curl --connect-time 10 --max-time 900 -ks https://splunk.mycompany.com:8089/services/search/jobs/export -umy_splunk_user:$p --data search='search earliest=-30m index=some-jenkins-prod-index source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on") | regex source=".*/[0-9][0-9]*/log" | table host, source, _raw' -d output_mode=json | jq ".result"
$
$ # The following will give you LINE by LINE output for the console output
$ curl --connect-time 10 --max-time 900 -ks https://splunk.mycompany.com:8089/services/search/jobs/export -umy_splunk_user:$p --data search='search earliest=-30m index=some-jenkins-prod-index source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on") | regex source=".*/[0-9][0-9]*/log" | table host, source, _raw' -d output_mode=json | jq -r ".result._raw"
注意： 用户 ID 和密码作为 -umy_splunk_user:$p 传递（不需要 space after/between -u 和实际的 splunk 用户名。
方案二：
解决方案编号。 2 是通过使用 SESSION KEY/ID。您首先必须使用 services/auth/login 终点。
生成SESSIONKEY/ID、运行以下命令。
注意： 要生成 SESSION 密钥，您需要先提供您的凭据，但在稍后的 CURL / API 调用/命令中，您可以在 header 中传递 SESSION 键。
1) 生成 session 密钥/id。
$ p=$(cat ~/my_secret_password.txt)
$ curl -k https://splunk.mycompany.com:8089/services/auth/login --data-urlencode username=my_splunk_userid --data-urlencode password=$p
<response>
<sessionKey>192fd3e46a31246da7ea7f109e7f95fd</sessionKey>
</response>
2) 以后在后续搜索中使用 session 键/ID。
在后续请求中，将 header 授权值设置为 session 密钥 () 并且 现在您不需要使用 传递您的凭据 -uYourUserID:YourPassword.
$ curl -k -H "Authorization: Splunk 192fd3e46a31246da7ea7f109e7f95fd" --connect-time 10 --max-time 900 https://splunk.mycompany.com:8089/services/search/jobs/export --data search='search earliest=-30m index=some-jenkins-prod-index source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on") | regex source=".*/[0-9][0-9]*/log" | table host, source, _raw' -d output_mode=json | jq ".result"
注意：
1) 对于控制台输出的逐行输出。使用：jq -r ".result._raw"
2) 对于找到的搜索计数，您可以使用 | stats count
现在，我可以得到我需要的 CSV 或 JSON 格式的数据，并使用图形功能通过有意义的图表显示数据，或者在阈值高于或低于给定阈值时发送电子邮件通知/ 期望值（根据我的自动化脚本）。
有关详细信息，请参阅 Splunk REST API 搜索端点文档：http://docs.splunk.com/Documentation/Splunk/6.6.3/RESTREF/RESTsearch
和
https://docs.splunk.com/Documentation/Splunk/6.5.3/SearchReference/SearchTimeModifiers
second: s, sec, secs, second, seconds
minute: m, min, minute, minutes
hour: h, hr, hrs, hour, hours
day: d, day, days
week: w, week, weeks
month: mon, month, months
quarter: q, qtr, qtrs, quarter, quarters
year: y, yr, yrs, year, years
如果您想搜索过去 30 天和之前 30 天的数据，您需要 earliest=-60d latest=-30d
解决方案#3：
1) 创建COOKIE ID，运行跟随命令。
curl -sSv https://splunk.mycompany.com:8089/services/auth/login --data-urlencode username=your_splunk_userid --data-urlencode password=your_splunk_secret_password -o /dev/null -d cookie=1 2>&1
它会吐出类似这样的东西：
< Set-Cookie: splunkd_8081=5omeJunk_ValueHere^kjadaf33999dasdx0ihe28gcEYvbP1yhTjcTjgQCRaOUhco6wwLf5YLsay_2JgZ^J^SEYF9f2nSYkyS0qbu_RE; Path=/; HttpOnly; Max-Age=28800; Expires=Wed, 20 Sep 2017 00:23:39 GMT
现在获取值部分 < Set-Cookie: <VALUE_upto_the_semi_colon> 并将其存储在一个变量中。即
export COOKIE="splunkd_8081=5omeJunk_ValueHere^kjadaf33999dasdx0ihe28gcEYvbP1yhTjcTjgQCRaOUhco6wwLf5YLsay_2JgZ^J^SEYF9f2nSYkyS0qbu_RE"
2) 现在，在您的 CURL 命令中使用 cookie 进行 运行ning 与我们上面所做的类似查询。您现在不需要 传递凭据-uYourUserID:Password。
$ curl -k --cookie "${COOKIE}" --connect-time 10 --max-time 900 ... rest of the command here similar to examples shown above ... ...
为了更好地实施，请执行以下操作：
Jenkins 的 Splunk 插件：
https://wiki.jenkins.io/display/JENKINS/Splunk+Plugin+for+Jenkins
Splunk 的 Jenkins Add-on/App：https://splunkbase.splunk.com/app/3332/
Splunk 版本 6.5.3.1
Splunk 构建 bf0ff7c2ab8b
Jenkins 版本 1.642.3 或 2.32.3
在每个 Jenkins master 上，都有一个 splunk 进程 运行ning。
$ ps -eAf|grep splunk
splunk 58877 1 20 Feb16 ? 42-23:27:37 splunkd -p 8089 restart
splunk 58878 58877 0 Feb16 ? 00:00:00 [splunkd pid=58877] splunkd -p 8089 restart [process-runner]
asangal 91197 91175 0 12:38 pts/2 00:00:00 grep --color=auto splunk
Splunk 进程监控/扫描 log 文件以查找我们实例中的任何 Jenkins 作业，即在 $JENKINS_HOME/jobs/<JOB_NAME>/builds/<BUILD_NUMBER>/log 文件下。
$ pwd
/opt/splunkforwarder/etc/system/local
$ cat inputs.conf
[default]
host = jenkins-master-project-prod-1-609
[monitor:///var/log/jenkins]
index = some-jenkins-prod-index-in-splunk
disabled = False
recursive = True
[monitor:///home/jenkins/jobs/.../builds/.../log]
index = some-jenkins-prod-index-in-splunk
disabled = False
recursive = True
crcSalt = <SOURCE>
...
.....
... more config code here ...
....
..
在 Splunk GUI 中，当我 运行 一个简单的查询来查找 Splunk 捕获的相同 index 并且来自任何 source （文件）的任何内容时，我确实看到有效的输出。 注意： 实际行输出是 t运行 满足的。正如您在条形图上看到的那样，数据在那里并且填充了 table。
在我的 Jenkins 工作中，我有时会收到一些警告、信息、错误（为此我已经在 Jenkins 级别使用 Log Parser Plugin），我正在尝试编写一个脚本来获取此日志Splunk 的 Jenkins 作业在过去 15、30 分钟或过去 1-7 小时或 1-30 天内的输出，并找出在给定时间段内发现了多少警告、错误等（基于某些关键字、正则表达式）。注意：有很多这样的 Jenkins masters，其中 Splunk 是 运行nings，我的目标是与 Splunk 对话并获取我需要的数据（而不是与 500 个 Jenkins masters 对话）。
我尝试了以下 CURL 命令，这些命令 returns 给我一个搜索 ID，但它没有做任何事情。
在下面的 CURL 命令中，我传递了一个更精确的查询来获取数据。我说的是在过去 30 分钟内获取 Splunk 拥有的所有信息（可以根据 GUI 添加字段），其中 index 是 some-jenkins-prod-index-in-splunk 并且日志的 source 是：/home/jenkins/jobs/*/builds/*/log（第一个 * 用于工作名称，第二个 * 用于内部版本号）然后我说在 splunk 中搜索日志，其中日志包含 lines/keywords/regex 之一（如下面使用 OR 条件列出）并以 JSON 格式显示输出。
➜ ~ p=$(cat ~/AKS/rOnly/od.p.txt)
➜ ~ curl --connect-time 10 --max-time 900 -ks https://splunk.server.mycompany.com:8089/services/search -umy_splunk_user:$p --data search='search earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on")' -d output_mode=json
{"messages":[{"type":"ERROR","text":"Method Not Allowed"}]}% ➜ ~
如您所见，它显示方法不允许。
当我在 URL 部分使用 /jobs 给出以下查询时，我得到了一个有效的搜索 ID。
➜ ~ curl --connect-time 10 --max-time 900 -ks https://splunk.server.mycompany.com:8089/services/search/jobs -umy_splunk_user:$p --data search='search earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on")' -d output_mode=json
{"sid":"1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A"}%
使用此搜索 ID，我试图访问主要日志，但它不起作用。我正在使用 jq 过滤 JSON 输出以显示在漂亮的布局中。
➜ ~ curl --connect-time 10 --max-time 900 -ks https://splunk.server.mycompany.com:8089/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A -umy_splunk_user:$p --data search='search earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on")' -d output_mode=json|jq .
{
"links": {},
"origin": "http://splunk.server.mycompany.com/services/search/jobs",
"updated": "2017-09-15T09:44:33-07:00",
"generator": {
"build": "bf0ff7c2ab8b",
"version": "6.5.3.1"
},
"entry": [
{
"name": "search earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log (\"WARNING: \" OR \"npm WARN retry\" OR \"svn: E200033: \" OR \": binary operator expected\" OR \": too many arguments\" OR \": No such file or directory\" OR \"rsync: failed to set times on\") | regex source=\".*/[0-9][0-9]*/log\" | table host, source, _raw",
"id": "http://splunk.server.mycompany.com/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A",
"updated": "2017-09-15T09:44:33.942-07:00",
"links": {
"alternate": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A",
"search.log": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/search.log",
"events": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/events",
"results": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/results",
"results_preview": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/results_preview",
"timeline": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/timeline",
"summary": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/summary",
"control": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/control"
},
"published": "2017-09-15T09:43:59.000-07:00",
"author": "my_splunk_user",
"content": {
"bundleVersion": "17557160226808436058",
"canSummarize": false,
"cursorTime": "1969-12-31T16:00:00.000-08:00",
"defaultSaveTTL": "2592000",
"defaultTTL": "600",
"delegate": "",
"diskUsage": 561152,
"dispatchState": "DONE",
"doneProgress": 1,
"dropCount": 0,
"earliestTime": "2017-09-15T09:13:58.000-07:00",
"eventAvailableCount": 0,
"eventCount": 30,
"eventFieldCount": 0,
"eventIsStreaming": true,
"eventIsTruncated": true,
"eventSearch": "search (earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log (\"WARNING: \" OR \"npm WARN retry\" OR \"svn: E200033: \" OR \": binary operator expected\" OR \": too many arguments\" OR \": No such file or directory\" OR \"rsync: failed to set times on\")) | regex source=\".*/[0-9][0-9]*/log\" ",
"eventSorting": "none",
"isBatchModeSearch": true,
"isDone": true,
"isEventsPreviewEnabled": false,
"isFailed": false,
"isFinalized": false,
"isPaused": false,
"isPreviewEnabled": false,
"isRealTimeSearch": false,
"isRemoteTimeline": false,
"isSaved": false,
"isSavedSearch": false,
"isTimeCursored": true,
"isZombie": false,
"keywords": "\"*: binary operator expected*\" \"*: no such file or directory*\" \"*: too many arguments*\" \"*npm warn retry*\" \"*rsync: failed to set times on*\" \"*svn: e200033: *\" \"*warning: *\" earliest::-30m index::some-jenkins-prod-index-in-splunk source::/home/jenkins/jobs/*/builds/*/log",
"label": "",
"latestTime": "2017-09-15T09:43:59.561-07:00",
"normalizedSearch": "litsearch ( index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log ( \"WARNING: \" OR \"npm WARN retry\" OR \"svn: E200033: \" OR \": binary operator expected\" OR \": too many arguments\" OR \": No such file or directory\" OR \"rsync: failed to set times on\" ) _time>=1505492038.000 ) | regex source=\".*/[0-9][0-9]*/log\" | fields keepcolorder=t \"_raw\" \"host\" \"source\"",
"numPreviews": 0,
"optimizedSearch": "| search (earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log (\"WARNING: \" OR \"npm WARN retry\" OR \"svn: E200033: \" OR \": binary operator expected\" OR \": too many arguments\" OR \": No such file or directory\" OR \"rsync: failed to set times on\")) | regex source=\".*/[0-9][0-9]*/log\" | table host, source, _raw",
"pid": "2174",
"priority": 5,
"remoteSearch": "litsearch ( index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log ( \"WARNING: \" OR \"npm WARN retry\" OR \"svn: E200033: \" OR \": binary operator expected\" OR \": too many arguments\" OR \": No such file or directory\" OR \"rsync: failed to set times on\" ) _time>=1505492038.000 ) | regex source=\".*/[0-9][0-9]*/log\" | fields keepcolorder=t \"_raw\" \"host\" \"source\"",
"reportSearch": "table host, source, _raw",
"resultCount": 30,
"resultIsStreaming": false,
"resultPreviewCount": 30,
"runDuration": 0.579,
"sampleRatio": "1",
"sampleSeed": "0",
"scanCount": 301,
"searchCanBeEventType": false,
"searchEarliestTime": 1505492038,
"searchLatestTime": 1505493839.21872,
"searchTotalBucketsCount": 37,
"searchTotalEliminatedBucketsCount": 0,
"sid": "1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A",
"statusBuckets": 0,
"ttl": 600,
"performance": {
"command.fields": {
"duration_secs": 0.035,
"invocations": 48,
"input_count": 30,
"output_count": 30
},
"command.regex": {
"duration_secs": 0.048,
"invocations": 48,
"input_count": 30,
"output_count": 30
},
"command.search": {
"duration_secs": 1.05,
"invocations": 48,
"input_count": 0,
"output_count": 30
},
"command.search.calcfields": {
"duration_secs": 0.013,
"invocations": 16,
"input_count": 301,
"output_count": 301
},
"dispatch.optimize.reparse": {
"duration_secs": 0.001,
"invocations": 1
},
"dispatch.optimize.toJson": {
"duration_secs": 0.001,
"invocations": 1
},
"dispatch.optimize.toSpl": {
"duration_secs": 0.001,
"invocations": 1
},
"dispatch.parserThread": {
"duration_secs": 0.048,
"invocations": 48
},
"dispatch.reduce": {
"duration_secs": 0.001,
"invocations": 1
},
"dispatch.stream.remote": {
"duration_secs": 1.05,
"invocations": 48,
"input_count": 0,
"output_count": 332320
},
"dispatch.stream.remote.mr11p01if-ztbv02090901.mr.if.mycompany.com-8081": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr11p01if-ztbv02090901.mr.if.mycompany.com-8082": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr11p01if-ztbv11204201.mr.if.mycompany.com-8081": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr11p01if-ztbv11204201.mr.if.mycompany.com-8082": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr11p01if-ztbv11204401.mr.if.mycompany.com-8081": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr11p01if-ztbv11204401.mr.if.mycompany.com-8082": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr11p01if-ztbv16142101.mr.if.mycompany.com-8081": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr11p01if-ztbv16142101.mr.if.mycompany.com-8082": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr11p01if-ztbv16142301.mr.if.mycompany.com-8081": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr11p01if-ztbv16142301.mr.if.mycompany.com-8082": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr21p01if-ztbv14080101.mr.if.mycompany.com-8081": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr21p01if-ztbv14080101.mr.if.mycompany.com-8082": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr22p01if-ztbv07132101.mr.if.mycompany.com-8081": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr22p01if-ztbv07132101.mr.if.mycompany.com-8082": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr22p01if-ztbv09013201.mr.if.mycompany.com-8081": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr22p01if-ztbv09013201.mr.if.mycompany.com-8082": {
"duration_secs": 0.001,
"invocations": 1,
"input_count": 0,
"output_count": 5422
},
"dispatch.stream.remote.mr90p01if-ztep02103701.mr.if.mycompany.com-8081": {
"duration_secs": 0.058,
"invocations": 2,
"input_count": 0,
"output_count": 16948
},
"dispatch.stream.remote.mr90p01if-ztep02103701.mr.if.mycompany.com-8082": {
"duration_secs": 0.066,
"invocations": 2,
"input_count": 0,
"output_count": 14415
},
"dispatch.stream.remote.mr90p01if-ztep04044101.mr.if.mycompany.com-8081": {
"duration_secs": 0.059,
"invocations": 2,
"input_count": 0,
"output_count": 15858
},
"dispatch.stream.remote.mr90p01if-ztep04044101.mr.if.mycompany.com-8082": {
"duration_secs": 0.065,
"invocations": 2,
"input_count": 0,
"output_count": 11867
},
"dispatch.stream.remote.mr90p01if-ztep06024101.mr.if.mycompany.com-8081": {
"duration_secs": 0.061,
"invocations": 2,
"input_count": 0,
"output_count": 20695
},
"dispatch.stream.remote.mr90p01if-ztep06024101.mr.if.mycompany.com-8082": {
"duration_secs": 0.06,
"invocations": 2,
"input_count": 0,
"output_count": 15193
},
"dispatch.stream.remote.mr90p01if-ztep12023601.mr.if.mycompany.com-8081": {
"duration_secs": 0.063,
"invocations": 2,
"input_count": 0,
"output_count": 15932
},
"dispatch.stream.remote.mr90p01if-ztep12023601.mr.if.mycompany.com-8082": {
"duration_secs": 0.064,
"invocations": 2,
"input_count": 0,
"output_count": 14415
},
"dispatch.stream.remote.mr90p01if-ztep12043901.mr.if.mycompany.com-8081": {
"duration_secs": 0.061,
"invocations": 2,
"input_count": 0,
"output_count": 15418
},
"dispatch.stream.remote.mr90p01if-ztep12043901.mr.if.mycompany.com-8082": {
"duration_secs": 0.058,
"invocations": 2,
"input_count": 0,
"output_count": 11866
},
"dispatch.stream.remote.pv31p01if-ztbv08050801.pv.if.mycompany.com-8081": {
"duration_secs": 0.075,
"invocations": 2,
"input_count": 0,
"output_count": 15661
},
"dispatch.stream.remote.pv31p01if-ztbv08050801.pv.if.mycompany.com-8082": {
"duration_secs": 0.071,
"invocations": 2,
"input_count": 0,
"output_count": 15845
},
"dispatch.stream.remote.pv31p01if-ztbv08051001.pv.if.mycompany.com-8081": {
"duration_secs": 0.066,
"invocations": 2,
"input_count": 0,
"output_count": 14406
},
"dispatch.stream.remote.pv31p01if-ztbv08051001.pv.if.mycompany.com-8082": {
"duration_secs": 0.072,
"invocations": 2,
"input_count": 0,
"output_count": 15524
},
"dispatch.stream.remote.pv31p01if-ztbv08051201.pv.if.mycompany.com-8081": {
"duration_secs": 0.067,
"invocations": 2,
"input_count": 0,
"output_count": 16009
},
"dispatch.stream.remote.pv31p01if-ztbv08051201.pv.if.mycompany.com-8082": {
"duration_secs": 0.068,
"invocations": 2,
"input_count": 0,
"output_count": 15516
},
"dispatch.writeStatus": {
"duration_secs": 0.012,
"invocations": 7
},
"startup.configuration": {
"duration_secs": 2.045,
"invocations": 33
},
"startup.handoff": {
"duration_secs": 14.595,
"invocations": 33
}
},
"messages": [
{
"type": "INFO",
"text": "Your timerange was substituted based on your search string"
},
{
"type": "WARN",
"text": "Unable to distribute to peer named pv31p01if-ztbv08050601.pv.if.mycompany.com:8081 at uri=pv31p01if-ztbv08050601.pv.if.mycompany.com:8081 using the uri-scheme=http because peer has status=\"Down\". Please verify uri-scheme, connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information."
},
{
"type": "WARN",
"text": "Unable to distribute to peer named pv31p01if-ztbv08050601.pv.if.mycompany.com:8082 at uri=pv31p01if-ztbv08050601.pv.if.mycompany.com:8082 using the uri-scheme=http because peer has status=\"Down\". Please verify uri-scheme, connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information."
}
],
"request": {
"search": "search earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log (\"WARNING: \" OR \"npm WARN retry\" OR \"svn: E200033: \" OR \": binary operator expected\" OR \": too many arguments\" OR \": No such file or directory\" OR \"rsync: failed to set times on\") | regex source=\".*/[0-9][0-9]*/log\" | table host, source, _raw"
},
"runtime": {
"auto_cancel": "0",
"auto_pause": "0"
},
"searchProviders": [
"mr11p01if-ztbv02090901.mr.if.mycompany.com-8081",
"mr11p01if-ztbv16142101.mr.if.mycompany.com-8082",
"mr11p01if-ztbv16142301.mr.if.mycompany.com-8081",
"mr11p01if-ztbv16142301.mr.if.mycompany.com-8082",
"mr21p01if-ztbv14080101.mr.if.mycompany.com-8081",
"mr21p01if-ztbv14080101.mr.if.mycompany.com-8082",
"mr22p01if-ztbv07132101.mr.if.mycompany.com-8081",
"mr22p01if-ztbv07132101.mr.if.mycompany.com-8082",
"mr22p01if-ztbv09013201.mr.if.mycompany.com-8081",
"mr22p01if-ztbv09013201.mr.if.mycompany.com-8082",
"mr90p01if-ztep02103701.mr.if.mycompany.com-8081",
"mr90p01if-ztep02103701.mr.if.mycompany.com-8082",
"mr90p01if-ztep04044101.mr.if.mycompany.com-8081",
"mr90p01if-ztep04044101.mr.if.mycompany.com-8082",
"mr90p01if-ztep06024101.mr.if.mycompany.com-8081",
"mr90p01if-ztep06024101.mr.if.mycompany.com-8082",
"mr90p01if-ztep12023601.mr.if.mycompany.com-8081",
"mr90p01if-ztep12023601.mr.if.mycompany.com-8082",
"mr90p01if-ztep12043901.mr.if.mycompany.com-8081",
"mr90p01if-ztep12043901.mr.if.mycompany.com-8082",
"pv31p01if-ztbv08050801.pv.if.mycompany.com-8081",
"pv31p01if-ztbv08050801.pv.if.mycompany.com-8082",
"pv31p01if-ztbv08051001.pv.if.mycompany.com-8081",
"pv31p01if-ztbv08051001.pv.if.mycompany.com-8082",
"pv31p01if-ztbv08051201.pv.if.mycompany.com-8081",
"pv31p01if-ztbv08051201.pv.if.mycompany.com-8082"
]
},
"acl": {
"perms": {
"read": [
"my_splunk_user"
],
"write": [
"my_splunk_user"
]
},
"owner": "my_splunk_user",
"modifiable": true,
"sharing": "global",
"app": "search",
"can_write": true,
"ttl": "600"
}
}
],
"paging": {
"total": 1,
"perPage": 0,
"offset": 0
}
}
➜ ~
➜ ~
但是，如您所见，生成的 JSON 输出没有用，因为它没有显示或包含我可以使用的任何 Jenkins 作业的输出。
如果在 CURL 命令中，对于 Splunk URL，如果我尝试以下任何 URLs 端点，它会给我一个错误。
"search.log": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/search.log",
"events": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/events",
"results": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/results",
"results_preview": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/results_preview",
"timeline": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/timeline",
"summary": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/summary",
"control": "/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/control"
例如： 如果我尝试 URL.../<SEARCH_ID>/events 或 URL/.../<SEARCH_ID>/results 等，我会收到以下错误。
curl --connect-time 10 --max-time 900 -ks https://splunk.server.mycompany.com:8089/services/search/jobs/1505493838.3723_ACEB82F4-AA21-4AE2-95A3-566F6BCAA05A/events -umy_splunk_user:$p --data search='search earliest=-30m index=some-jenkins-prod-index-in-splunk source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on")' -d output_mode=json|jq .
{
"messages": [
{
"type": "FATAL",
"text": "Method Not Allowed"
}
]
}
我正在尝试查找主机名、来源（Jenkins 作业日志的路径）、实际作业的控制台输出（我可以读取和解析以生成有意义的信息）关于最近 N 个时间段，有多少，出现错误、警告、奇怪的线条，并且取决于某些阈值，如果数字超过这些阈值，那么我需要发送电子邮件通知。
我可以编写所有这些代码，但我没有得到这里的第一个难题，即让 Splunk 吐出 Jenkins 作业的控制台输出，其中 splunk 在文件系统上监视。
最终目标是将有意义的数据以 JSON 或 CSV 格式转储到文本文件中，并将该数据转换为一些有意义的 bar/pie 图表等。
例如：如果 data.csv 包含：
age,population
<5,2704659
5-13,4499890
14-17,2159981
18-24,3853788
25-44,14106543
45-64,8819342
65-85,312463
≥85,81312463
然后使用以下文件，我可以将此原始数据转换为饼图，如下图所示。
<!DOCTYPE html>
<meta charset="utf-8">
<style>
.arc text {
font: 10px sans-serif;
text-anchor: middle;
}
.arc path {
stroke: #fff;
}
</style>
<svg width="960" height="500"></svg>
<script src="https://d3js.org/d3.v4.min.js"></script>
<script>
var svg = d3.select("svg"),
width = +svg.attr("width"),
height = +svg.attr("height"),
radius = Math.min(width, height) / 2,
g = svg.append("g").attr("transform", "translate(" + width / 2 + "," + height / 2 + ")");
var color = d3.scaleOrdinal(["#98abc5", "#8a89a6", "#7b6888", "#6b486b", "#a05d56", "#d0743c", "#ff8c00"]);
var pie = d3.pie()
.sort(null)
.value(function(d) { return d.population; });
var path = d3.arc()
.outerRadius(radius - 10)
.innerRadius(0);
var label = d3.arc()
.outerRadius(radius - 40)
.innerRadius(radius - 40);
d3.csv("data.csv", function(d) {
d.population = +d.population;
return d;
}, function(error, data) {
if (error) throw error;
var arc = g.selectAll(".arc")
.data(pie(data))
.enter().append("g")
.attr("class", "arc");
arc.append("path")
.attr("d", path)
.attr("fill", function(d) { return color(d.data.age); });
arc.append("text")
.attr("transform", function(d) { return "translate(" + label.centroid(d) + ")"; })
.attr("dy", "0.35em")
.text(function(d) { return d.data.age; });
});
</script>
生成的饼图（由于csv文件和html文件）：
找到解决方案。
我只需要使用 services/search/jobs/export 终点。
让我们找出什么是我们的 Jenkins host（其中包含 Jenkins 作业），作业的名称是什么（这可以从日志文件的 source 路径中解析/搜索）以及 Jenkins 作业的实际控制台输出是什么 (_raw)。此外，让我们将搜索限制在最近 30 分钟内（即 earliest=-30m）。
实际上有3种方法可以做到。
1) 通过在命令行传递用户名和密码。
2) 通过生成一个 SESSION TOKEN，我们可以在 header 后的任何未来 CURL 命令中使用它。
3) 通过生成 --cookie "${COOKIE}" ID 并使用它。这是所有方法中的首选方法，因为它将 cookie 值复制到 Splunk 使用的任何后端服务器。要使用的 Cookie 名称：splunkd_8081
后两种解决方案取决于第一种方法，即使用用户凭据创建 SESSION 或 COOKIE ID。
解决方案一：
1) 这里我们将使用我们的 splunk 服务器
2) 在命令行中传递用户名和密码
3) 提供用于查找/获取 Splunk 数据的 Splunk 选项（对于包含特定行的 Jenkins 日志），并进行一些额外的正则表达式匹配（这样它将 return 与源的 Jenkins build# 完全一致路径而不是为相同的控制台输出显示 source 的 3 行。Jenkins latestBuild、latestSuccessfulBuild 等是指向编号构建的符号链接，我们不想在我们的列表中列出这些符号链接的 source 条目输出，所以我使用正则表达式模式来查找 source 路径，其中它在源路径中的 log 文件之前包含一个 NUMBERED build#。
4) 然后，我使用 | 仅过滤掉 3 个字段：host、source 和 _raw（Splunk 会 return). host 包含，哪个 Jenkins 服务器有 Jenkins 作业。 source 的值中包含 Jenkins 作业名称、build# 等信息。 _raw 变量包含 Jenkins 作业的控制台输出（靠近我们试图在 Jenkins 作业的控制台输出中搜索的字符串/行的几行）。
注意： 所有这 3 个字段都在字典变量中可用 result 所以我只是输出它。
5) 然后，我以 json 格式提供输出（您也可以使用 csv）。最后我使用 jq 来过滤信息。
注意： 如果您使用 jq -r ".result._raw"（即字典变量 result 中的 _raw 字段，那么它将逐行显示控制台输出的输出（而不是给你一个嵌入了 \n 的 blob）。你也可以使用 sed 's/\n/\n/g' 但是 jq -r ".result._raw" 很简单够了）。
命令 运行:
$ p="$(cat ~/my_secret_password.txt)"
$
$ # The above command will set my password in variable 'p'
$
$ curl --connect-time 10 --max-time 900 -ks https://splunk.mycompany.com:8089/services/search/jobs/export -umy_splunk_user:$p --data search='search earliest=-30m index=some-jenkins-prod-index source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on") | regex source=".*/[0-9][0-9]*/log" | table host, source, _raw' -d output_mode=json | jq ".result"
$
$ # The following will give you LINE by LINE output for the console output
$ curl --connect-time 10 --max-time 900 -ks https://splunk.mycompany.com:8089/services/search/jobs/export -umy_splunk_user:$p --data search='search earliest=-30m index=some-jenkins-prod-index source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on") | regex source=".*/[0-9][0-9]*/log" | table host, source, _raw' -d output_mode=json | jq -r ".result._raw"
注意： 用户 ID 和密码作为 -umy_splunk_user:$p 传递（不需要 space after/between -u 和实际的 splunk 用户名。
方案二：
解决方案编号。 2 是通过使用 SESSION KEY/ID。您首先必须使用 services/auth/login 终点。
生成SESSIONKEY/ID、运行以下命令。
注意： 要生成 SESSION 密钥，您需要先提供您的凭据，但在稍后的 CURL / API 调用/命令中，您可以在 header 中传递 SESSION 键。
1) 生成 session 密钥/id。
$ p=$(cat ~/my_secret_password.txt)
$ curl -k https://splunk.mycompany.com:8089/services/auth/login --data-urlencode username=my_splunk_userid --data-urlencode password=$p
<response>
<sessionKey>192fd3e46a31246da7ea7f109e7f95fd</sessionKey>
</response>
2) 以后在后续搜索中使用 session 键/ID。
在后续请求中，将 header 授权值设置为 session 密钥 () 并且 现在您不需要使用 传递您的凭据 -uYourUserID:YourPassword.
$ curl -k -H "Authorization: Splunk 192fd3e46a31246da7ea7f109e7f95fd" --connect-time 10 --max-time 900 https://splunk.mycompany.com:8089/services/search/jobs/export --data search='search earliest=-30m index=some-jenkins-prod-index source=/home/jenkins/jobs/*/builds/*/log ("WARNING: " OR "npm WARN retry" OR "svn: E200033: " OR ": binary operator expected" OR ": too many arguments" OR ": No such file or directory" OR "rsync: failed to set times on") | regex source=".*/[0-9][0-9]*/log" | table host, source, _raw' -d output_mode=json | jq ".result"
注意：
1) 对于控制台输出的逐行输出。使用：jq -r ".result._raw"
2) 对于找到的搜索计数，您可以使用 | stats count
现在，我可以得到我需要的 CSV 或 JSON 格式的数据，并使用图形功能通过有意义的图表显示数据，或者在阈值高于或低于给定阈值时发送电子邮件通知/ 期望值（根据我的自动化脚本）。
有关详细信息，请参阅 Splunk REST API 搜索端点文档：http://docs.splunk.com/Documentation/Splunk/6.6.3/RESTREF/RESTsearch 和 https://docs.splunk.com/Documentation/Splunk/6.5.3/SearchReference/SearchTimeModifiers
second: s, sec, secs, second, seconds
minute: m, min, minute, minutes
hour: h, hr, hrs, hour, hours
day: d, day, days
week: w, week, weeks
month: mon, month, months
quarter: q, qtr, qtrs, quarter, quarters
year: y, yr, yrs, year, years
如果您想搜索过去 30 天和之前 30 天的数据，您需要 earliest=-60d latest=-30d
解决方案#3：
1) 创建COOKIE ID，运行跟随命令。
curl -sSv https://splunk.mycompany.com:8089/services/auth/login --data-urlencode username=your_splunk_userid --data-urlencode password=your_splunk_secret_password -o /dev/null -d cookie=1 2>&1
它会吐出类似这样的东西：
< Set-Cookie: splunkd_8081=5omeJunk_ValueHere^kjadaf33999dasdx0ihe28gcEYvbP1yhTjcTjgQCRaOUhco6wwLf5YLsay_2JgZ^J^SEYF9f2nSYkyS0qbu_RE; Path=/; HttpOnly; Max-Age=28800; Expires=Wed, 20 Sep 2017 00:23:39 GMT
现在获取值部分 < Set-Cookie: <VALUE_upto_the_semi_colon> 并将其存储在一个变量中。即
export COOKIE="splunkd_8081=5omeJunk_ValueHere^kjadaf33999dasdx0ihe28gcEYvbP1yhTjcTjgQCRaOUhco6wwLf5YLsay_2JgZ^J^SEYF9f2nSYkyS0qbu_RE"
2) 现在，在您的 CURL 命令中使用 cookie 进行 运行ning 与我们上面所做的类似查询。您现在不需要 传递凭据-uYourUserID:Password。
$ curl -k --cookie "${COOKIE}" --connect-time 10 --max-time 900 ... rest of the command here similar to examples shown above ... ...
为了更好地实施，请执行以下操作：
Jenkins 的 Splunk 插件： https://wiki.jenkins.io/display/JENKINS/Splunk+Plugin+for+Jenkins
Splunk 的 Jenkins Add-on/App：https://splunkbase.splunk.com/app/3332/