What is the best practice to deal with equal stanzas in input.conf due to wildcards?

[sha_knowis]

Hey everybody,

we have some problems with our inputs.conf for directory inputs in the following stanzas:
[monitor:///pathToLogs/*/fixedPath/logForSourcetype1*.log]
[monitor:///pathToLogs/*/fixedPath/logForSourcetype2*.log]

The goal here is to read the host and source type for the given input.
- host: through host_segment (first * in the stanzas)
- source type: through the name of the logfile(logForSourctype[1/2])

Our problem is, that as defined in the documentation, a monitor with wildcards gets separated into the monitor and a whitelist.

Therefore the stanzas will look like:

[monitor:///pathToLogs/]
whitelist = [^/]*/fixedPath/logForSourcetype1[^/]*\.log

and

[monitor:///pathToLogs/]
whitelist = [^/]*/fixedPath/logForSourcetype2[^/]*\.log

(see: http://docs.splunk.com/Documentation/Splunk/7.1.3/Data/Specifyinputpathswithwildcards#Wildcards_and_...)

As a result, both stanzas are equal and differ only in the whitelist.

Therefore the second stanza will overwrite the first, which can also be seen in the _internal logs.

We found a solution for equal stanzas in another Splunk question.

The proposition for equal stanzas and different sourcetypes was to define the sourcetype in props.conf through source.
(see: https://answers.splunk.com/answers/2692/3-monitor-stanzas-of-the-same-folder-but-only-one-sourcetype...)

However, the post was tailored for 4.1 and we would be interested to know if there were a better and more elegant solution for our problem.

[sloshburch]

I believe the post you referenced still stands as the strongest approach.

Use inputs.conf to collect the most generic pattern: monitor:///pathToLogs/*/fixedPath/logForSourcetype*.log (Notice the number is now wildcarded in the filename).

And then use a sourcetype and host override to assign those fields dynamically depending on the source matches.
- Override source types on a per-event basis
- Set host values based on event data

You may choose to assign the host and sourcetype to silly values as a way to ensure the health of this config. For example:

[monitor:///pathToLogs/*/fixedPath/logForSourcetype*.log]
host = changeMe
sourcetype = changeMe

And then you can have an alert for any events that appear with host=changeMe OR sourcetype=changeMe so you become aware when your configuration is failing.

[sudosplunk]

Hi,

You can assign a sourcetype name to whatever name you'd like. Can you see if below inputs.conf works for you,

[monitor:///pathToLogs/*/fixedPath/logForSourcetype1*.log]
host_segment = 2
sourcetype = logForSourcetype1
index = your_index

[monitor:///pathToLogs/*/fixedPath/logForSourcetype2*.log]
host_segment = 2
sourcetype = logForSourcetype2
index = your_index

[frank_buettner]

But as @sha_knowis mentioned, a monitor containing a wildcard gets converted into a monitor with an absolute file input path and a whitelist.
See [documentation][2]:

When you specify wildcards in a file input path, Splunk Enterprise creates an implicit whitelist for that stanza. The longest wildcard-free path becomes the monitor stanza, and Splunk Enterprise translates the wildcards into regular expressions.

So your example will result in two monitors with the file input path ///pathToLogs/ with different whitelists. But the file input paths of monitors must be unique. If not, the last monitor in inputs.conf wins.