All Apps and Add-ons

Jenkins data indexing into Splunk, dashboards all blank

eandresen
Path Finder

I am testing out the Splunk App for Jenkins (v.1.0.7) on my Proof of Concept instance of Splunk (v6.6.0), but I am having problems with it working properly, specially the field extractions. The downloaded app is checked into our Git repo, which pushes it out to the Splunk Search Cluster Deployer and down the a 4-node Search Cluster using Splunk's built in deployment processes.

As shown below, I have setup both the Jenkins plugin and Splunk app, which is feeding in plenty of data under the four indexes created under the app. Looking through the related Jenkin events, everything looks fine to me.

alt text

However, when I switch over to the dashboards within the app, every panel is completely blank. The only thing that seems to be working is the "Jenkins Master" dropdown option within each dashboard page.

alt text

After doing some digging, I have found that none of the custom field extractions that come with the app are working. I only see the normal index, sourcetype, souce, host, splunk_server, etc. key/value pairs when I run searches from the dashboards. An example search string and all of the resulting fields returned are provided below.

  • Search string: index=jenkins_statistics (host="jenkins-etsb.tools.expedia.com" )
  • host
  • index
  • source
  • sourcetype
  • linecount
  • splunk_server

I also noticed that the only field extraction showing in the UI is the "source::jenkins://... : EXTRACT-username" or the first one in the props.conf file under the app. I even commented out the entire "[source::jenkins://...]" stanza to see if it was causing problems, than nothing showed up in the fields extraction.

The really strange part is when I run the btool command against the Splunk App for Jenkins app and the props.conf settings, everything from the props.conf file appear to be loaded without issues. (And, yes I have both restarted the Splunk service and ran a debug-refresh command several times).

[root@ip-10-0-18-18 ~]# /opt/splunk/bin/splunk cmd btool props list --app=splunk_app_jenkins
[json:jenkins]
INDEXED_EXTRACTIONS = json
KV_MODE = none
LEARN_MODEL = false
TRUNCATE = 0
category = Structured
description = JavaScript Object Notation format.
pulldown_type = true
[json:jenkins:old]
KV_MODE = json
LEARN_MODEL = false
TRUNCATE = 0
category = Structured
description = JavaScript Object Notation format.
pulldown_type = true
[source::.../job_event]
EVAL-build_plan = if(type="completed" AND isnull('test_summary.total'), job_name, NULL)
EVAL-duration = duration * 1000
EVAL-fail_count = 'test_summary.failures'
EVAL-id = coalesce(id, build_url)
EVAL-pass_count = 'test_summary.passes'
EVAL-skip_count = 'test_summary.skips'
EVAL-status = case(job_result=="SUCCESS", "Successful", job_result=="FAILURE", "Failed", job_result=="UNSTABLE", "Unstable", job_result=="ABORTED", "Aborted")
EVAL-test_suite_id = if(type="completed" AND isnotnull('test_summary.total'), job_name, NULL)
EVAL-testless = if(isnull('test_summary.total'), 1, 0)
[source::jenkins://...]
EXTRACT-username = ^\<!--\<!\[CDATA\[(?P<username>.*?)\]\]\>--\>\n(?P<jenkins_config>.*)
KV_MODE = xml
[text:jenkins]
LEARN_MODEL = false
SHOULD_LINEMERGE = false
TRUNCATE = 100000
category = Miscellaneous
pulldown_type = true
0 Karma

Gattaca2
Explorer

We had the same issue, and in the end it was because we had the HEC Jenkins was pushing data to on a heavy forwarder, so the props.conf needs installing on there as well.

Regards
Andy

eandresen
Path Finder

Picture of search results...

alt text

0 Karma

aakwah
Builder

Thanks for providing the snapshot, can you confirm that the search mode is set to verbose not fast?

0 Karma

aakwah
Builder

At my end I've fields extracted as per this snapshot: (Splunk ver. 6.5.3)

https://drive.google.com/open?id=0B9wUSHOfDLvoM3ZYT0RXZGF1Y0E

Regards

0 Karma

eandresen
Path Finder

It was ran in smart mode...

0 Karma

aakwah
Builder

You have mentioned that you have deployed the app via Search head cluster deployer which is fine, but you need to copy indexes.conf and props.conf from default folder on the app to indexers and in case of indexer cluster to this directory $SPLUNK_HOME/etc/master-apps/_cluster/local/ on cluster master, then run "splunk apply cluster-bundle".

This is because indexed extractions need to be applied during indexing time.

as per props.conf doc (http://docs.splunk.com/Documentation/Splunk/6.6.0/Admin/Propsconf)

 Structured Data Header Extraction and configuration

 * This feature and all of its settings apply at input time, when data is
   first read by Splunk.  The setting is used on a Splunk system that has
   configured inputs acquiring the data.

Regards

0 Karma

aakwah
Builder

Hello,

Regarding field extraction, it is handled by INDEXED_EXTRACTIONS = json in [json:jenkins] stanza that converts json object into key /value pairs.

I had a quick look on the app and I that found most of search queries of the dashboards contain event_tag=job_event so if incoming events don't have event_tag field with "job_event" value then dashboards will be empty, so I believe that you need to generate some real data on your Proof of Concept.

Run this query to check what event_tag's you have at the moment:

index=jenkins* | stats count by event_tag

Hope this helps.

Regards

0 Karma

eandresen
Path Finder

Thanks for the reply!

The Jenkins job related events has an event_tag field; however, it is one of the many fields that are not being extracted because all of the props.conf extractions are not working. I will add a picture of what I mean in another answer since I cannot attached it here or add more to the original post.

0 Karma

aakwah
Builder

You have mentioned that you have deployed the app via Search head cluster deployer which is fine, but you need to copy indexes.conf and props.conf from default folder on the app to indexers and in case of indexer cluster to this directory $SPLUNK_HOME/etc/master-apps/_cluster/local/ on cluster master, then run "splunk apply cluster-bundle".

This is because indexed extractions need to be applied during indexing time.

as per props.conf doc (http://docs.splunk.com/Documentation/Splunk/6.6.0/Admin/Propsconf)

Structured Data Header Extraction and configuration

* This feature and all of its settings apply at input time, when data is
  first read by Splunk.  The setting is used on a Splunk system that has
  configured inputs acquiring the data.

Regards

0 Karma

eandresen
Path Finder

Yep, that was already done per the instructions that came with the app. I copied both files onto the Index Master and it deployed down the Indexer Cluster. That is why the jenkins* indexes exist and data is being indexed under them. With that said, this is the first time I have worked with Indexer Clustering, so I am not completely sure the deployment process worked properly. Though, like the index settings on the Indexers and the props settings on the Search Cluster, the btool command returns everything I would expect.

[root@ip-10-0-18-125 ~]# /opt/splunk/bin/splunk cmd btool indexes list --debug --app=splunk_app_jenkins_indexers
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf [jenkins]
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf coldPath = volume:splunk_all/jenkins/colddb
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf frozenTimePeriodInSecs = 2592000
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf homePath = volume:splunk_all/jenkins/db
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf repFactor = auto
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf thawedPath = $SPLUNK_DB/jenkins/thaweddb
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf [jenkins_artifact]
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf coldPath = volume:splunk_all/jenkins_artifact/colddb
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf frozenTimePeriodInSecs = 2592000
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf homePath = volume:splunk_all/jenkins_artifact/db
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf repFactor = auto
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf thawedPath = $SPLUNK_DB/jenkins_artifact/thaweddb
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf [jenkins_console]
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf coldPath = volume:splunk_all/jenkins_console/colddb
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf frozenTimePeriodInSecs = 2592000
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf homePath = volume:splunk_all/jenkins_console/db
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf repFactor = auto
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf thawedPath = $SPLUNK_DB/jenkins_console/thaweddb
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf [jenkins_statistics]
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf coldPath = volume:splunk_all/jenkins_statistics/colddb
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf frozenTimePeriodInSecs = 2592000
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf homePath = volume:splunk_all/jenkins_statistics/db
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf repFactor = auto
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf thawedPath = $SPLUNK_DB/jenkins_statistics/thaweddb
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf [volume:splunk_all]
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf maxVolumeDataSizeMB = 3400000
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/indexes.conf path = $SPLUNK_DB

[root@ip-10-0-18-125 ~]# /opt/splunk/bin/splunk cmd btool props list --debug --app=splunk_app_jenkins_indexers
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf [json:jenkins]
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf INDEXED_EXTRACTIONS = json
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf KV_MODE = none
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf LEARN_MODEL = false
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf TRUNCATE = 0
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf category = Structured
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf description = JavaScript Object Notation format.
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf pulldown_type = true
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf [json:jenkins:old]
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf KV_MODE = json
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf LEARN_MODEL = false
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf TRUNCATE = 0
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf category = Structured
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf description = JavaScript Object Notation format.
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf pulldown_type = true
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf [source::.../job_event]
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf EVAL-build_plan = if(type="completed" AND isnull('test_summary.total'), job_name, NULL)
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf EVAL-duration = duration * 1000
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf EVAL-fail_count = 'test_summary.failures'
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf EVAL-id = coalesce(id, build_url)
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf EVAL-pass_count = 'test_summary.passes'
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf EVAL-skip_count = 'test_summary.skips'
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf EVAL-status = case(job_result=="SUCCESS", "Successful", job_result=="FAILURE", "Failed", job_result=="UNSTABLE", "Unstable", job_result=="ABORTED", "Aborted")
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf EVAL-test_suite_id = if(type="completed" AND isnotnull('test_summary.total'), job_name, NULL)
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf EVAL-testless = if(isnull('test_summary.total'), 1, 0)
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf [source::jenkins://...]
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf EXTRACT-username = ^\<!--\<!\[CDATA\[(?P<username>.*?)\]\]\>--\>\n(?P<jenkins_config>.*)
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf KV_MODE = xml
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf [text:jenkins]
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf LEARN_MODEL = false
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf SHOULD_LINEMERGE = false
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf TRUNCATE = 100000
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf category = Miscellaneous
/opt/splunk/etc/slave-apps/splunk_app_jenkins_indexers/default/props.conf pulldown_type = true
0 Karma

taldavita
Explorer

Was this resolved? I'm running into the same issues.
I've a heavy forwarder collecting the events, forwarding to two load balanced indexers.

The events are indexed however not parsed as json in the search head. Only the basic fields are shown.

Search head and Indexers are v.6.5.2 - Yes they've been restarted after installing. the props.conf is on the indexers.

This query returns no results
index=jenkins_statistics (host="sea1l1esbfb01" ) event_tag=job_event (type=started OR type=completed) | dedup host build_url sortby -_time | eval job_result=if(type="started", "INPROGRESS", job_result) | timechart count by job_result

However this query does
index=jenkins_statistics (host="sea1l1esbfb01" ) | spath | search event_tag=job_event (type=started OR type=completed) | dedup host build_url sortby -_time | eval job_result=if(type="started", "INPROGRESS", job_result) | timechart count by job_result

0 Karma
Get Updates on the Splunk Community!

Splunk is Nurturing Tomorrow’s Cybersecurity Leaders Today

Meet Carol Wright. She leads the Splunk Academic Alliance program at Splunk. The Splunk Academic Alliance ...

Part 2: A Guide to Maximizing Splunk IT Service Intelligence

Welcome to the second segment of our guide. In Part 1, we covered the essentials of getting started with ITSI ...

Part 1: A Guide to Maximizing Splunk IT Service Intelligence

As modern IT environments continue to grow in complexity and speed, the ability to efficiently manage and ...