Running into a strange scenario with host field resolution in combination with Splunk App for Stream. All the forwarders are pushed to end-points using chef and they currently copy over the inputs.conf from /etc/system/default
into /etc/system/local
adding a couple of monitor stanzas to the one in local to monitor additional files. This carries over the default stanza with host=$decideOnStartup
into the inputs.conf under local as well
[default]
index = default
_rcvbuf = 1572864
host = $decideOnStartup
The monitor inputs however, do get the resolved (correct) host information, as per below. Once the Splunk App for Stream is pushed to the forwarders, the stream data though picks up the host as $decideOnStartup
unless either the host is explicitly set in the inputs.conf under /etc/system/local
or the forwarder is cleaned up, and no inputs.conf from /etc/system/default
is copied over to /etc/system/local
which then effectively means the forwarder creates one at startup with the resolved hostname under which then keeps things humming along.
Question is, how are the monitor inputs able to resolve the host=$decideOnStartup
to the correct host name, even when placed under /etc/system/local
while the Stream input is not? Anyone seen this before and have suggestions on how to get past it, without having to either (a) manually set the host field or (b) redeploy the forwarders without a copy of the inputs.conf from /etc/system/default
https://answers.splunk.com/answers/3184/hostname-in-inputs-conf.html
By default, Splunk creates $SPLUNK_HOME/etc/system/local/inputs.conf on first time run. You're overwriting that by copying $SPLUNK_HOME/etc/system/default/inputs.conf to $SPLUNK_HOME/etc/system/local/inputs.conf. Now, this should work, but it doesn't. I filed a bug, SPL-108269 to track it, but will probably be a while before anyone gets to it given pretty valid workarounds available. This will affect any modular input you're using, not just Stream.
My suggestion to you is to put your configurations you want to monitor in your own app, or at a the very least, put them in $SPLUNK_HOME/etc/apps/search/local/inputs.conf and not change the default of having your hostname determined at First Time Run and written to $SPLUNK_HOME/etc/system/local/inputs.conf.
https://answers.splunk.com/answers/3184/hostname-in-inputs-conf.html
By default, Splunk creates $SPLUNK_HOME/etc/system/local/inputs.conf on first time run. You're overwriting that by copying $SPLUNK_HOME/etc/system/default/inputs.conf to $SPLUNK_HOME/etc/system/local/inputs.conf. Now, this should work, but it doesn't. I filed a bug, SPL-108269 to track it, but will probably be a while before anyone gets to it given pretty valid workarounds available. This will affect any modular input you're using, not just Stream.
My suggestion to you is to put your configurations you want to monitor in your own app, or at a the very least, put them in $SPLUNK_HOME/etc/apps/search/local/inputs.conf and not change the default of having your hostname determined at First Time Run and written to $SPLUNK_HOME/etc/system/local/inputs.conf.
Thanks Clint! Will work on tweaking the inputs per the note above
What are you modifying from /etc/system/default/inputs.conf and why are you copying the file wholesale rather than specifying only the parameters you want to override?
Adding a monitor stanza to monitor some additional files. This is an inherited system where the current set-up pushes out a copy of the /etc/system/default/inputs.conf under /etc/system/local. I understand that this isn't the best practice, but looking to see how to fix this problem short of having to blow away the whole UF and reinstall or having to hardcode the host