Here is a quick python script for an example of how to query this data and return it back to Splunk.
DISCLAIMER - This is NOT an officially Splunk supported function and merely just an example of how I get Jenkins data in for my own use in Splunk.
import requests, json
import collections
import splunk.Intersplunk as si
import time
import re
import sys
# replace YOURJENKINSURL with your Jenkins location
baseurl="http://YOURJENKINSURL:8080/"
# grab the arguments to be passed to the API
args=sys.argv[1:]
# if no arguments are given, list all jobs
if len(sys.argv)==1:
url = baseurl+"/api/json?depth=1"
# else handle projects, jobs, and builds
elif "project" in sys.argv:
project=sys.argv[sys.argv.index("project")+1]
if "job" in sys.argv:
if "build" in sys.argv:
job=sys.argv[sys.argv.index("job")+1]
build=sys.argv[sys.argv.index("build")+1]
url = baseurl+"job/"+project+"/job/"+job+"/"+build+"/api/json"
else:
job=sys.argv[sys.argv.index("job")+1]
url = baseurl+"job/"+project+"/job/"+job+"/api/json"
else:
url = baseurl+"job/"+project+"/api/json"
else:
url = baseurl+sys.argv[1]
# send request, don't verify SSL
r = requests.get(url,verify=False)
# if additional args weren't specified, just give the jobs otherwise use whole payload
if len(sys.argv)==1:
jdata=r.json()['jobs']
else:
jdata=r.json()
# quick and dirty conversion from dicts to json
def convert(data):
if isinstance(data, basestring):
return str(data)
elif isinstance(data, collections.Mapping):
return dict(map(convert, data.iteritems()))
elif isinstance(data, collections.Iterable):
return type(data)(map(convert, data))
else:
return data
class mydict(dict):
def __str__(self):
return json.dumps(self)
# initialize our results and rows
results=[]
row={}
# if no args given just deliver the jobs
if len(sys.argv)==1:
for i in jdata:
# this sets Splunk's _time as the time of the script run
row["_time"]=time.time()
# arbitrary Splunk host field assignment, change as desired
row["host"]="Jenkins"
# arbitrary Splunk source field assignment, change as desired
row["source"]="Jenkins"
# arbitrary Splunk sourcetype field assignment, change as desired
row["sourcetype"]="Jenkins"
# create an _raw field (to show as events in Splunk) using our dict to json conversion
row["_raw"]=mydict(convert(i))
# add row to result
results.append(row)
# clear out the row since we've appended it. We want this empty so we don't end up with just the last row of data
row={}
# when the results are complete, stream them on to Splunk
si.outputStreamResults(results)
else:
# if we have args, parse accordingly
try:
# this sets Splunk's _time as the time of the script run
row["_time"]=time.time()
# arbitrary Splunk host field assignment, change as desired
row["host"]="Jenkins"
# arbitrary Splunk source field assignment, change as desired
row["source"]="Jenkins"
# arbitrary Splunk sourcetype field assignment, change as desired
row["sourcetype"]="Jenkins"
# assign fields based on args
if "project" in sys.argv:
row["project"]=project
if "job" in sys.argv:
row["job"]=job
if "build" in sys.argv:
row["build"]=build
# create an _raw field (to show as events in Splunk) using our dict to json conversion
row["_raw"]=mydict(convert(jdata))
# add row to result
results.append(row)
# clear out the row since we've appended it. We want this empty so we don't end up with just the last row of data
row={}
# when the results are complete, stream them on to Splunk
si.outputStreamResults(results)
# some very minor error handling that will return an error event back to Splunk if the API call fails
except Exception, e:
import traceback
results=[]
row={}
stack = traceback.format_exc()
row['_time'] = int(time.time())
row['error'] = str(str(e))
results.append(row)
si.outputStreamResults(results)
logger.error(str(e) + ". Traceback: " + str(stack))
Save the above as a python script (for example, jenkins.py) then you'll need to create a commands.conf for wherever you put the command (could be in its own app, or what have you, but wherever you put it, make sure the commands.conf exists there). The commands.conf should exist in the "local" directory of the App you put the script in.
# identify the command name
[jenkins]
# tell Splunk what the script name is on disk
filename = jenkins.py
# we're making our own time fields, so just ignore the order that the events come in
overrides_timeorder = true
# some values may be multivalues, we should support it
supports_multivalues = true
# streaming should be set to true since we're using InterSplunks outputStreamResults
streaming = true
# it's a generating command
generating = stream
# send the sessionKey when the command is run (probably extraneous, sends SessionKey to the command)
passauth = true
# don't try to run this distributed (if you set local to false, you will run the same command over all search heads getting duplicate data!)
local = true
And finally, you'll want a props.conf in the "local" directory of the App you put the command in. This helps Splunk to know how to extract the fields returned.
# identify the sourcetype (IMPORTANT! if you changed the sourcetype in the script, make sure you change it here too!)
[Jenkins]
# use json extractions
INDEXED_EXTRACTIONS = json
# ignore kv_mode
KV_MODE = none
You'll want to restart after you've gotten all the files in place. ( You ARE using a test system for this first right? Don't dev in Prod! 😉 )
Search syntax for Jenkins -
| jenkins project [project name] job [job name] build [build name]{/testReport}|spath
You can do each individually if you like
| jenkins project [project name] |spath
| jenkins project [project name] job [job name] |spath
| jenkins project [project name] job [job name] build [build name]|spath
/testReport is case sensitive and must immediately follow the build name
Note that all data is returned in JSON, it's highly recommended to use |spath for field extraction
If it's not working as expected, try a couple of things first -
Go to the Jenkin's API URI -
http://YOURJENKINS:8080/api/json?depth=1
Run the command manually from the command line (shell) using Splunk's python interpreter.
/opt/splunk/bin/splunk cmd python jenkins.py
... View more
