All Apps and Add-ons

Jenkins data sourcetype mapping

D2SI
Communicator

Hello there,

I am trying to setup the App.

So far, I have:

  • installed the App on the Search Head
  • created the dedicated indexes
  • installed props.conf on the Heavy Forwarder

  • installed the plugin on a jenkins master

  • configured the plugin to establish connection with the HEC

  • configured the plugin to send console log & build events

I have a JSON parsing issues but I believe it is because I have set console log with "json:jenkins" sourcetype while I guess it is supposed to be "test:jenkins" instead.

Anyhow, that leads to my question:

It is not really clear to me how to map data (Step 2 Configure Metadata).

How the table below could be filled?

Event type_Index_Sourcetype

Build Report_index?_sourcetype?
Build Event_index?_sourcetype?
Queue Information_index: jenkins_statistics_sourcetype?
Console Log_index: jenkins_console_sourcetype: text:jenkins
Log File_index: jenkins_artifact_sourcetype?
Slave Information_index: jenkins_statistics_sourcetype?
Jenkins Config_index?_sourcetype?

Indexes:

jenkins
jenkins_statistics
jenkins_console
jenkins_artifact

Sourcetypes:

json:jenkins
text:jenkins

2nd question:

How many HEC inputs should be configured on the Heavy Forwarder?

I mean, it seems that you can allow multiple indexes for 1 HEC input. But 1 HEC input is tied to 1 sourcetype, right?

So I guess I would need 2 HEC inputs, 1 tied to 'json:jenkins' the other to 'text:jenkins' and configure tokens on the jenkins plugin accordingly.

Thanks in advance for any help,

0 Karma
1 Solution

txiao_splunk
Splunk Employee
Splunk Employee

The default plugin config should work out of box for the the App, unless you are using a highly customized app

Per the doc https://wiki.jenkins.io/display/JENKINS/Splunk+Plugin+for+Jenkins
Metadata configuration for Splunk App for Jenkins
For Splunk version 6.5 or later, it is recommended to use the plugin's default config
For Splunk 6.3.x or 6.4.x, please adjust the default sourcetype to json:jenkins:old (please remove it if Splunk get upgraded to latest version otherwise data will be extracted twice)
alt text

View solution in original post

txiao_splunk
Splunk Employee
Splunk Employee

The default plugin config should work out of box for the the App, unless you are using a highly customized app

Per the doc https://wiki.jenkins.io/display/JENKINS/Splunk+Plugin+for+Jenkins
Metadata configuration for Splunk App for Jenkins
For Splunk version 6.5 or later, it is recommended to use the plugin's default config
For Splunk 6.3.x or 6.4.x, please adjust the default sourcetype to json:jenkins:old (please remove it if Splunk get upgraded to latest version otherwise data will be extracted twice)
alt text

View solution in original post

D2SI
Communicator

Hello txiao,

Sorry I have been confused and it is actually simpler than I thought it would be. Even with a default configuration, the plugin takes care of it all and data is being indexed to various indexes and sourcetypes. It is all alright.

Thanks for the help!

0 Karma

D2SI
Communicator

Anyone ?

0 Karma

arkadyz1
Builder

Looking at Jenkins documentation, it seems like you can configure any sourcetype there. I believe an HEC input will accept any source, sourcetype, host, time etc. if they are specified in the incoming message. The only restriction is that the index (if it is set in the incoming message) should be in the list of allowed indexes (configure the token correspondingly).

Read the section named "Event Metadata" in "Format Events for HTTP event collector" within "Getting Data In" manual. There is no restriction on sourcetype there.

0 Karma

D2SI
Communicator

Hello arkadyz1,

I also understand the documentation this way : you can set up any data source with any index / sourcetype you want.

Thing is the App has the indexes hardcoded.

For instance when looking for the 'jenkins_console' string (one of the suggested indexes) in the App files, it founds multiple occurrences in multiple javascript files used in the App.

So I feel like I need to figure out the mapping to make sure the App has the data it needs in the dedicated indexes.

Regarding the sourcetype, I do not know beforehand which one is text, which on is json.

Thanks anyhow!

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.