Why is the data not getting displayed in Splunk from Jenkins?

[bwaj]

I am getting the below error in Jenkins after adding Splunk plugin, and data is also not getting uploaded to Splunk from Jenkins.

May 10, 2018 11:06:20 AM com.splunk.splunkjenkins.utils.LogConsumer run
SEVERE: message not delivered:{length:1687 {"host":"","index":"jenkins_statistics","sourcetype":"json:jenkins","time":"1525946780.049","source":"jenkins","event":{"queue_time" ...}
com.splunk.splunkjenkins.utils.LogConsumer$SplunkClientError: Bad Request, incorrect index or invalid data format, status code:400
    at com.splunk.splunkjenkins.utils.LogConsumer$1.handleResponse(LogConsumer.java:68)
    at com.splunk.splunkjenkins.utils.LogConsumer$1.handleResponse(LogConsumer.java:40)
    at shaded.splk.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:222)
    at shaded.splk.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:164)
    at shaded.splk.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:139)
    at com.splunk.splunkjenkins.utils.LogConsumer.run(LogConsumer.java:90)

[txiao_splunk]

the HEC token doesn't have write access to jenkins_statistics, please config the token
http://dev.splunk.com/view/event-collector/SP-CAAAE7F

[neluvasilica]

Since I've upgraded Splunk plugin on one of the masters I am seeing the same errors(logs are flooded with them) but only on specific builds:

Mar 25, 2020 4:43:51 PM com.splunk.splunkjenkins.utils.LogConsumer run
SEVERE: message not delivered:{length:62626 {"host":"xxxxxx","index":"jenkins_console","sourcetype":"text:jenkins","time":"1585154629.437","source":"job/xxxxxxx/job/master/55 ...}
com.splunk.splunkjenkins.utils.LogConsumer$SplunkClientError: Bad Request, incorrect index or invalid data format, status code:400
        at com.splunk.splunkjenkins.utils.LogConsumer$1.handleResponse(LogConsumer.java:68)
        at com.splunk.splunkjenkins.utils.LogConsumer$1.handleResponse(LogConsumer.java:40)
        at shaded.splk.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:223)
        at shaded.splk.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165)
        at shaded.splk.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:140)
        at com.splunk.splunkjenkins.utils.LogConsumer.run(LogConsumer.java:89)

Any idea what could be causing this?

[neluvasilica]

I've actually noticed this happening on other masters too and it is generating error messages every second or so which is impacting the Jenkins servers' performance.
@txiao_splunk has anyone else reported this issue with the latest version of the Splunk plugin?

[txiao_splunk]

looks like a config issue.
the HEC token is either incorrect or does not have write access to jenkins_console, or jenkins_console index doesn't exist at all

[MuS]

Did you blank the "host":"" information or is it actually missing?

cheers, MuS

[bwaj]

Hi Mus,

I am not sure of this option, can you help me to find where it will be?

[xpac]

Does the index jenkins_statistics exist on your system?

[bwaj]

Yes, I see the INDEX present on my Splunk server and its in enabled state also.