All Apps and Add-ons

Splunk App for Jenkins: After configuration, why are not all the tabs in the app showing data?

erosch
Explorer

Sorry for the long post but I need some help 😄

After watching the Splunking Jenkins presentation, we started setting the Splunk App for Jenkins with half success.

We have a Splunk cloud instance, so Splunk Support installed the Splunk App for Jenkins for us. After configuring everything the best we could, in the app, some sections have no results, some sections do.

What works:
In all pages, I can select the "host" in the "Jenkins Master" box.
Overview > Master/Slave Topology works!
Jenkins Health works!
Jenkins Slave works!

What doesn't work:
Overview > Jenkins Build Status History and Latest Builds shows "No results found"
Build Analysis > I can only select the "host" in the "Jenkins Master" box. Every other box, I can only select "*", again, it says "No results found"
Audit Trails shows "No results found" in both sections

Somewhat works:
Test Results > I can see results, but they are not correct. (For some builds that are unstable (test failed), it only shows the passing tests, but not the failing ones. I am using python's nosetests which outputs a xunit format report which Jenkins publishes)

Here is our configuration...

1. Collector and token setup by support
Support gave us the event collector URL and a token. We tested the token against the indexes:
curl -k [collectorURL] -H 'Authorization: Splunk [token]' -d '{"event":"test message", "index":"jenkins"}'
curl -k [collectorURL] -H 'Authorization: Splunk [token]' -d '{"event":"test message", "index":"jenkins_console"}'
curl -k [collectorURL] -H 'Authorization: Splunk [token]' -d '{"event":"test message", "index":"jenkins_artifact"}'
curl -k [collectorURL] -H 'Authorization: Splunk [token]' -d '{"event":"test message", "index":"jenkins_statistics"}'

In splunk search, we run: index=jenkins* "test message" |table _raw, index
All good at this point, we got all the 4 "test message".

(Note: when contacting support, they created a HEC and asked for the index and sourcetype for it. We said index=debug, sourcetype=xml. I don't know if this is relevant, but the information was not clear in the Jenkins plugin documentation about what to say to support, now it seems as if we don't specify the index in the metadata, the default goes to "debug" and we get a funny error if metadata is simply host=mymasterhost)

2. Jenkins configuration
We entered our splunk host, port=443, the token given by support, and our Jenkins MetaData is:

sourcetype=_json
source=jenkins
host=masterhostname

#custom index, uncomment the settings after index created in Splunk
index=jenkins_artifact
build_report.index=jenkins
file.index=jenkins_artifact
console_log.index=jenkins_console
queue_info.index=jenkins_statistics
slave_info.index=jenkins_statistics

jenkins_config.enabled=true
jenkins_config.monitoring=true

3. Jobs
We did not change our jobs.

Am I missing anything? I am also confused why Test Results doesn't show me the right results (fail tests doesn't show up). Funny enough, sometimes it does show me failed tests, so it's not consistent, I haven't figured out a pattern yet... if someone can help me with this as well...

1 Solution

txiao_splunk
Splunk Employee
Splunk Employee

From the symptom

View solution in original post

txiao_splunk
Splunk Employee
Splunk Employee

From the symptom

erosch
Explorer

Thanks, we were using the old jenkins plugin, updating to the latest sent the data we were missing to Splunk App. So now results are showing.

For tests not showing up, I used the Search to see if the test reports events were being sent correctly to Splunk. It seems the "build_report" events are being cut at 10,000 characters (it's one long line). When our tests fails, it outputs a lot of information, which makes the event very long. I am guessing this is what is causing the test results to show partly. Is this something we can control in Jenkins plugin? I searched questions online that I have to change the props.conf. I am a Splunk Cloud user. Please advice.

This our current Event Processing command:
//send job metadata and junit reports with page size set to 50 (each event contains max 50 test cases)
def results = getJunitReport()
def buildEvent = getBuildEvent()
results.eachWithIndex { junitResult, idx ->
Map pagedEvent = buildEvent + ["testsuite": junitResult, "page_num": idx + 1]
send(pagedEvent)
}

0 Karma

txiao_splunk
Splunk Employee
Splunk Employee

Please remove the "sourcetype=_json" in jenkins metadata config to let the plugin to use default sourcetype "json:jenkins" for build report

The App shipped a props.conf with below content

[json:jenkins]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = none
TRUNCATE = 0
category = Structured
description = JavaScript Object Notation format.

[json:jenkins:old]
pulldown_type = true
TRUNCATE = 0
#INDEXED_EXTRACTIONS was not supported in HTTP Event Collector prior to 6.5.0
KV_MODE = json
category = Structured
description = JavaScript Object Notation format.

[text:jenkins]
pulldown_type = true
SHOULD_LINEMERGE=false
TRUNCATE = 100000
category = Miscellaneous

Please ask support to install the app or the props.conf in the splunk instance which is providing the HTTP Event Collector function.

You can also install a Heavy Forwarder with HTTP Event Collector enabled, and update the props.conf on Heavy Forwarder if you want to manage the Splunk yourself. Please refer https://docs.splunk.com/Documentation/Splunk/6.5.0/Forwarding/Deployaheavyforwarder , your splunk cloud instance will be the receiver.

by the way, if the code def results = getJunitReport() is not a paste error, please change it to def results = getJunitReport(50) or even a smaller page size def results = getJunitReport(30)

0 Karma

erosch
Explorer

Long test results are showing complete now.

I changed sourcetype=_json to sourcetype=json:jenkins and i am using def results = getJunitReport(30)

def results = getJunitReport(30) made the test results be sent as multiple events. The event size is around 40,000 characters now.
I think sourcetype=json:jenkins made it apply the settings in props.conf because it is not truncating anymore.

Also did extra testing on the jenkins plugin, so if I write sourcetype=aaa, click Save, then edit again by deleting it, it doesn't set it to a default sourcetype=json:jenkins, new events will be sent with sourcetype=aaa (last one specified). Looks like a jenkins plugin bug if a default was expected. So the best way was to be explicit in metadata.

0 Karma

aaraneta_splunk
Splunk Employee
Splunk Employee

@erosch - Did txiao answer your question? If yes, please don't forget to click "Accept" below their answer. If not, please provide some more feedback. Thank you.

0 Karma

txiao_splunk
Splunk Employee
Splunk Employee

sourcetype=aaa was cached and confirmed as a plugin bug, thanks for reporting.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...