How do I monitor Jenkins job logs and junitResult.xml from the same directory tree and parse them differently?

[mszebenyi_splun]

I’m setting up Splunk for our jenkins logs which are in a directory tree as that looks like this

Splunk Branch/
    <Job>/
        <date>/
            junitResult.xml
            job.log
    <Job>/
        <date>/
            etc…

I’ve set up a file monitor at the top but it wants to parse the XML files in the tree as raw text which doesn’t work out so well. Is there a way to have the filemon determine the source type of each file on the fly to parse it correctly? Thanks.

[jensonthottian]

What metrics were you able to derive from the xml files. Will you be able to share some screenshots.

[somesoni2]

Have two different monitoring stanzas like this (assuming file extensions will be constant)

[monitor:///Splunk branch/*/*/*.xml]
your xml file data input configuration
index
sourcetype

[monitor:///Splunk branch/*/*/*.log]
your log file data input configuration
index
sourcetype

[hexx · Accepted solution]

Well, you really have two options here:

• Create two overlapping file monitor stanzas to selectively target and sourcetype:

  [monitor:///src/splunk/current///*.xml]
  sourcetype = xml
  index = poopies

  [monitor:///src/splunk/current///*.log]
  sourcetype = logs
  index = unicorns

This will allow you to set up two sourcetype-based stanzas in props.conf to configure event processing for each of these sourcetypes individually:

[xml]
BREAK_ONLY_BEFORE = <\?xml version=
MAX_EVENTS = 10000
(...)

[logs]
SHOULD_LINEMERGE = false
(...)

... or ...

• Have an all-encompassing file monitor stanza, and create two source-based stanzas in props.conf to configure event processing for each of these sourcetypes individually.

In inputs.conf:

[monitor:///src/splunk/current]

In props.conf, this would look like:

[source:/src/splunk/current/.../*.xml]
sourcetype = xml
index = poopies
BREAK_ONLY_BEFORE = <\?xml version=
MAX_EVENTS = 10000

[source:/src/splunk/current/.../*.log]
sourcetype = logs
index = unicorns
SHOULD_LINEMERGE = false

For the record, my preference goes to the first solution, mainly because I really don't like to see file monitor stanzas without a sourcetype and index specified. The best practice there is to set those two properties in inputs.conf as much as possible.