Problem in indexing different sourcetypes from jenkins build artifcats - XML and CSV

[viramamo]

Hi,
I need to send files(ex: XML and CSV) from Jenkins to Splunk.
For which,
I have integrated Jenkins and Splunk using below,
1) Jenkins app in Splunk(Splunk app for Jenkins) and
2) Splunk plug-in in Jenkins

In Jenkins,
I have configured "Custom Metadata" like this
[Data Source->'logfile', Config Item-> 'Source Type', Value->'custom_xml_sourcetype' ]
[Data Source->'logfile', Config Item-> 'Source Type', Value->'csv' ]

In Splunk,
Indexed files(ex: a.xml and b.csv) has the same sourcetype ->'custom_xml_sourcetype'. Ideally both needs to be mapped to the exact sourcetypes. Which should be like this,

a.xml -> 'custom_xml_sourcetype'
b.csv->'csv'

but, it is currently mapped like this,

a.xml -> 'custom_xml_sourcetype'
b.csv->'custom_xml_sourcetype'

It is not certain, how to map the sourcetypes to the correct file.
Also, There is no job level config information available as well.

My Requirements is very simple to get different files generated from the jenkins build artifact to splunk with different sourcetypes.
Where the Jenkins is configured as Master -> Slave setup.

Kindly suggest me it is possible to use the Jenkin's splunk plug-in app or go with different approach.

Many thanks.

[viramamo · Accepted solution]

Indexing of different sourcetypes such as .xml to Splunk from Jenkins is not possible, but indexing key:value pair in format of .csv or .json is possible again with its own limitations.

This is due to HEC in splunk, which listens to only one sourcetype also it needs to be in key:value pair. Jenkin's Splunk Plugin is an agent which forwards the data by parsing the files in jenkin's system, if the data of the file is of key:value pair, then they will be sent in the events key in the Json paylod to HEC ex: {"events":"key:value"}.

But what if the data of the file is not key:value pair ex: xml, then there is no guarantee that the exact key:value pair will be extracted out of the xml. Due to which the data extraction by the jenkin's splunk plugin agent will be failed for non key:value pair sourcetypes.

Basically, HEC of Splunk has limited usecase. The communication between Jenkins to Splunk happens through this will not be useful for indexing all the files with different file format.

Solution:
There needs to be conversion scripts in jenkins side to convert all the non key:value pairs to key:value pair. This will help in indexing appropriate data into splunk, but again it cannot be mapped to more than one sourcetype.