Getting Data In

What is the best practice to deal with equal stanzas in input.conf due to wildcards?

sha_knowis
New Member

Hey everybody,

we have some problems with our inputs.conf for directory inputs in the following stanzas:
[monitor:///pathToLogs/*/fixedPath/logForSourcetype1*.log]
[monitor:///pathToLogs/*/fixedPath/logForSourcetype2*.log]

The goal here is to read the host and source type for the given input.
- host: through host_segment (first * in the stanzas)
- source type: through the name of the logfile(logForSourctype[1/2])

Our problem is, that as defined in the documentation, a monitor with wildcards gets separated into the monitor and a whitelist.

Therefore the stanzas will look like:

[monitor:///pathToLogs/]
whitelist = [^/]*/fixedPath/logForSourcetype1[^/]*\.log

and

[monitor:///pathToLogs/]
whitelist = [^/]*/fixedPath/logForSourcetype2[^/]*\.log

(see: http://docs.splunk.com/Documentation/Splunk/7.1.3/Data/Specifyinputpathswithwildcards#Wildcards_and_...)

As a result, both stanzas are equal and differ only in the whitelist.

Therefore the second stanza will overwrite the first, which can also be seen in the _internal logs.

We found a solution for equal stanzas in another Splunk question.

The proposition for equal stanzas and different sourcetypes was to define the sourcetype in props.conf through source.
(see: https://answers.splunk.com/answers/2692/3-monitor-stanzas-of-the-same-folder-but-only-one-sourcetype...)

However, the post was tailored for 4.1 and we would be interested to know if there were a better and more elegant solution for our problem.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

I believe the post you referenced still stands as the strongest approach.

Use inputs.conf to collect the most generic pattern: monitor:///pathToLogs/*/fixedPath/logForSourcetype*.log (Notice the number is now wildcarded in the filename).

And then use a sourcetype and host override to assign those fields dynamically depending on the source matches.
- Override source types on a per-event basis
- Set host values based on event data

You may choose to assign the host and sourcetype to silly values as a way to ensure the health of this config. For example:

[monitor:///pathToLogs/*/fixedPath/logForSourcetype*.log]
host = changeMe
sourcetype = changeMe

And then you can have an alert for any events that appear with host=changeMe OR sourcetype=changeMe so you become aware when your configuration is failing.

sudosplunk
Motivator

Hi,

You can assign a sourcetype name to whatever name you'd like. Can you see if below inputs.conf works for you,

[monitor:///pathToLogs/*/fixedPath/logForSourcetype1*.log]
host_segment = 2
sourcetype = logForSourcetype1
index = your_index

[monitor:///pathToLogs/*/fixedPath/logForSourcetype2*.log]
host_segment = 2
sourcetype = logForSourcetype2
index = your_index
0 Karma

frank_buettner
Explorer

But as @sha_knowis mentioned, a monitor containing a wildcard gets converted into a monitor with an absolute file input path and a whitelist.
See [documentation][2]:

When you specify wildcards in a file input path, Splunk Enterprise creates an implicit whitelist for that stanza. The longest wildcard-free path becomes the monitor stanza, and Splunk Enterprise translates the wildcards into regular expressions.

So your example will result in two monitors with the file input path ///pathToLogs/ with different whitelists. But the file input paths of monitors must be unique. If not, the last monitor in inputs.conf wins.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...