Splunk App for Jenkins: Why do I have to log into the indexers in order to see data returned in my dashboards?

[mikefettis]

I have the Splunk App for Jenkins installed on my search heads, indexers, and heavy forwarders. It appears that the only place that anything is getting data back in the dashboards is when i log into the indexers and use the app there.

[mikefettis · Accepted solution]

The heavy forwarders that jenkins was connected to were running splunk 6.3.

The search heads and indexers were all running 6.5.
After I upgraded the heavy forwarders to 6.5 the search heads began rendering the jenkins dashboards properly.

[txiao_splunk]

what is the sourcetype for the event?

index="jenkins_statistics" job_event |table sourcetype,_raw

if you are using latest jenkins plugin with default settings, it should be json:jenkins, which is defined in the app as

[json:jenkins]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = none
TRUNCATE = 0
category = Structured
description = JavaScript Object Notation format.

If the splunk http event collector instance version is prior to 6.5.0, can you try modify jenkins config to use sourcetype json:jenkins:old? it can be done via

Advanced->Custom Metadata and choose values like below:
Data Source: Default
Config Item: Source Type
Value: json:jenkins:old

if you are using splunk 6.5.x, can you check the bundle replication log on search head for errors

index="_internal" source="*/splunkd.log" DistributedBundleReplicationManager

[gjanders]

This sounds like either you do not have access to the relevant indexes via the search head or your search heads are not retrieving the data from the required indexer servers.

Have you clicked on the search icon/magnifying glass within a dashboard panel and checked if the search works within the search head?

If you cannot see any data try searching to see if you have access to data from the required index:

| tstats count groupby index

To be more specific you can do:

| tstats count where index=<thejenkinsindexgoeshere> groupby index

And see if you obtain results, if not we just need to determine why you cannot access that index...

[mikefettis]

running from the search head , and the query "| tstats count where index=jenkins_statistics groupby index" i get back plenty of data so

i know the search heads can access the indexes it almost seemed to me like an event extraction isn't working properly. I looked at some of the searches that the dashboard had generated
this search returns nothing "index=jenkins_statistics event_tag=job_event (type=started OR type=completed)" but if i tear it down to just
"index=jenkins_statistics job_event " i can see events

[gjanders]

Interesting, if you do view the events within the jenkins_statistics do you see the fields tag, event_tag or type ?

If not then the issue is straightforward.