All Apps and Add-ons

Splunk App for Jenkins missing certain console lines for a Jenkins Job Builder job

Contributor

We're using the Splunk App for Jenkins and Splunk Plugin for Jenkins in order to ingest Jenkins logs into Splunk. By default, this app / plugin combo provides console logs to index=jenkins_console sourcetype=text:jenkins. By and large this is working as expected. However, we are seeing some Jenkins builds that are simply forwarding blank lines for console lines that are not empty when you review them in Jenkins.

I've looked at other Splunk Answers, but wasn't able to find another question that had our same setup. I know that Jenkins Pipeline code needs to explicitly call a sendSplunkConsoleLog function (e.g., https://answers.splunk.com/answers/691730/how-to-get-jenkins-console-into-splunk.html). However, we are not using Jenkins Pipeline. We are using Jenkins Job Builder.

Here is an example of what we are seeing when running a search like index=jenkins_console sourcetype=text:jenkins:

Console output in Splunk

...
2019-10-08T17:45:11.517-0400 line:6 [WS-CLEANUP] Deleting project workspace...
2019-10-08T17:45:11.526-0400 line:7 Run condition [Boolean condition] enabling prebuild for step [Trigger/call builds on other projects]
2019-10-08T17:45:11.527-0400 line:8 Run condition [Not] preventing prebuild for step [Windows PowerShell]
2019-10-08T17:45:11.560-0400 line:9
2019-10-08T17:45:12.286-0400 line:10
2019-10-08T17:45:12.737-0400 line:11
2019-10-08T17:45:13.623-0400 line:12
2019-10-08T17:45:13.695-0400 line:13
2019-10-08T17:45:13.696-0400 line:14 items.find({
2019-10-08T17:45:13.696-0400 line:15 "@build.name": "F5-master",
2019-10-08T17:45:13.696-0400 line:16 "repo": "F5-master",
2019-10-08T17:45:13.696-0400 line:17 "@gitBranch": "master",
2019-10-08T17:45:13.696-0400 line:18 "@project.name": "F5ManagementClient-master-Tool"
...

Actual console output in Jenkins

...
17:45:11 [WS-CLEANUP] Deleting project workspace...
17:45:11 Run condition [Boolean condition] enabling prebuild for step [Trigger/call builds on other projects]
17:45:11 Run condition [Not] preventing prebuild for step [Windows PowerShell]
17:45:11 [CompareGroupsAppFamily-staging-idleDeploy-qwest] $ powershell.exe -NonInteractive -ExecutionPolicy ByPass "& 'C:\Users\Hudson\AppData\Local\Temp\jenkins8757820116422196227.ps1'"
17:45:12 ###### start macro - shared-splunk-createCustomMetadataLog
17:45:12 [CompareGroupsAppFamily-staging-idleDeploy-qwest] $ powershell.exe -NonInteractive -ExecutionPolicy ByPass "& 'C:\Users\Hudson\AppData\Local\Temp\jenkins5520032045158571584.ps1'"
17:45:13 ###### start macro - shared-artifactory-downloadLatestBuild
17:45:13 #################### Query To Be Invoked ####################
17:45:13 items.find({
17:45:13     "@build.name":  "F5-master",
17:45:13     "repo":  "F5-master",
17:45:13     "@gitBranch":  "master",
17:45:13     "@project.name":  "F5ManagementClient-master-Tool"
...

As you can see from above, Splunk recognizes that lines 9 through 13 (i.e., the stuff above items.find) exist, but Splunk only receives them as empty lines. When looking at the output in Jenkins, you can see that the lines are not empty and in fact contain crucial information for us.

Does anyone know why this might be happening? Please let me know if you need any more information.

0 Karma

Splunk Employee
Splunk Employee

the argument lines were discarded,

[workspace-name] launcher-name(powershell/bash) arguments

it may relate to https://issues.jenkins-ci.org/browse/JENKINS-52729
I verified on Jenkins 2.190.3 and it fine (1.642.4 isn't), Could you please try latest Jenkins LTS version https://jenkins.io/doc/upgrade-guide/?

0 Karma

Contributor

@txiao - Thanks for the comment. I'm a little confused where the line of code you mentioned exists. Is that a configuration in Splunk or Jenkins?

Specifically, I'm wondering where you found this:
[workspace-name] launcher-name(powershell/bash) arguments

0 Karma

Motivator

How many lines of console output are being generated from your Jenkins build?

0 Karma

Contributor

It seems to be a little under 200 lines on average. I looked at one of the most recent ones and it is 189 lines.

0 Karma

Motivator

Hrmm ok. Well, I had a possibly theory but its ruled out by that.
I had noticed in the text:jenkins sourcetype there is a TRUNCATE setting of 100k lines.

I took the console data you provided above and indexed it with the text:jenkins sourcetype though.
The results I get returned from a search are identical to the raw logs.

The only idea I can think of now is that your props.conf / sourcetype is potentially either missing or has incorrect permissions and your events are not parsing correctly.

0 Karma

Motivator

Have you checked _internal for any warnings/errors related to the jenkins_console index, parsing issues, etc.?

0 Karma