Some events in splunk do not correspond to Jenkins Console output.

[schilds427]

Hey folks,

I suspect this is more of a Jenkins problem than a Splunk problem, but I figured I'd ask here anyway. Our Jenkins instances have the 'Splunk App for Jenkins' installed and set to send all console output from all jobs to Splunk on the "text::jenkins" sourcetype. Recently though I've noticed that there seem to be more events in Splunk than there should be. For example if I do a search to look for a specific job's console output , like

index=jenkins_console sourcetype="text:jenkins" earliest=-5d@d latest=now source=source_of_job_I_Want | stats values(_raw) by _time

I see: 

From Splunk query

But if I go directly to Jenkins and look at the console output for this job I see: From Jenkins Console

This is an issue because we have an alert for the "Cannot contact X: java.lang.InterruptedException" String to help detect agent failures, but it's confusing when that alert goes off but then we look at the console and don't see that message. 

Anyone come across anything similar? 

[richgalloway]

Is the possible the message was generated by the TA rather than by Jenkins?

[schilds427]

Like from the 'Splunk for Jenkins' plugin? I don't think so. Looking at the configuration for the plugin it appears to be pretty much the default settings: