All Apps and Add-ons
Highlighted

Why is the Splunk App for Stream input tagging host as $decideOnStartup while monitor inputs resolve correctly?

Path Finder

Running into a strange scenario with host field resolution in combination with Splunk App for Stream. All the forwarders are pushed to end-points using chef and they currently copy over the inputs.conf from /etc/system/default into /etc/system/local adding a couple of monitor stanzas to the one in local to monitor additional files. This carries over the default stanza with host=$decideOnStartup into the inputs.conf under local as well

[default]
index         = default
_rcvbuf        = 1572864
host = $decideOnStartup

The monitor inputs however, do get the resolved (correct) host information, as per below. Once the Splunk App for Stream is pushed to the forwarders, the stream data though picks up the host as $decideOnStartup unless either the host is explicitly set in the inputs.conf under /etc/system/local or the forwarder is cleaned up, and no inputs.conf from /etc/system/default is copied over to /etc/system/local which then effectively means the forwarder creates one at startup with the resolved hostname under which then keeps things humming along.

Question is, how are the monitor inputs able to resolve the host=$decideOnStartup to the correct host name, even when placed under /etc/system/local while the Stream input is not? Anyone seen this before and have suggestions on how to get past it, without having to either (a) manually set the host field or (b) redeploy the forwarders without a copy of the inputs.conf from /etc/system/default

Tags (2)
0 Karma
Highlighted

Re: Why is the Splunk App for Stream input tagging host as $decideOnStartup while monitor inputs resolve correctly?

Communicator

What are you modifying from /etc/system/default/inputs.conf and why are you copying the file wholesale rather than specifying only the parameters you want to override?

0 Karma
Highlighted

Re: Why is the Splunk App for Stream input tagging host as $decideOnStartup while monitor inputs resolve correctly?

Path Finder

Adding a monitor stanza to monitor some additional files. This is an inherited system where the current set-up pushes out a copy of the /etc/system/default/inputs.conf under /etc/system/local. I understand that this isn't the best practice, but looking to see how to fix this problem short of having to blow away the whole UF and reinstall or having to hardcode the host

0 Karma
Highlighted

Re: Why is the Splunk App for Stream input tagging host as $decideOnStartup while monitor inputs resolve correctly?

Communicator

https://answers.splunk.com/answers/3184/hostname-in-inputs-conf.html

By default, Splunk creates $SPLUNKHOME/etc/system/local/inputs.conf on first time run. You're overwriting that by copying $SPLUNKHOME/etc/system/default/inputs.conf to $SPLUNK_HOME/etc/system/local/inputs.conf. Now, this should work, but it doesn't. I filed a bug, SPL-108269 to track it, but will probably be a while before anyone gets to it given pretty valid workarounds available. This will affect any modular input you're using, not just Stream.

My suggestion to you is to put your configurations you want to monitor in your own app, or at a the very least, put them in $SPLUNKHOME/etc/apps/search/local/inputs.conf and not change the default of having your hostname determined at First Time Run and written to $SPLUNKHOME/etc/system/local/inputs.conf.

View solution in original post

Highlighted

Re: Why is the Splunk App for Stream input tagging host as $decideOnStartup while monitor inputs resolve correctly?

Path Finder

Thanks Clint! Will work on tweaking the inputs per the note above

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.