Running into a strange scenario with host field resolution in combination with Splunk App for Stream. All the forwarders are pushed to end-points using chef and they currently copy over the inputs.conf from
/etc/system/local adding a couple of monitor stanzas to the one in local to monitor additional files. This carries over the default stanza with
host=$decideOnStartup into the inputs.conf under local as well
[default] index = default _rcvbuf = 1572864 host = $decideOnStartup
The monitor inputs however, do get the resolved (correct) host information, as per below. Once the Splunk App for Stream is pushed to the forwarders, the stream data though picks up the host as
$decideOnStartup unless either the host is explicitly set in the inputs.conf under
/etc/system/local or the forwarder is cleaned up, and no inputs.conf from
/etc/system/default is copied over to
/etc/system/local which then effectively means the forwarder creates one at startup with the resolved hostname under which then keeps things humming along.
Question is, how are the monitor inputs able to resolve the
host=$decideOnStartup to the correct host name, even when placed under
/etc/system/local while the Stream input is not? Anyone seen this before and have suggestions on how to get past it, without having to either (a) manually set the host field or (b) redeploy the forwarders without a copy of the inputs.conf from
What are you modifying from /etc/system/default/inputs.conf and why are you copying the file wholesale rather than specifying only the parameters you want to override?
Adding a monitor stanza to monitor some additional files. This is an inherited system where the current set-up pushes out a copy of the /etc/system/default/inputs.conf under /etc/system/local. I understand that this isn't the best practice, but looking to see how to fix this problem short of having to blow away the whole UF and reinstall or having to hardcode the host
By default, Splunk creates $SPLUNKHOME/etc/system/local/inputs.conf on first time run. You're overwriting that by copying $SPLUNKHOME/etc/system/default/inputs.conf to $SPLUNK_HOME/etc/system/local/inputs.conf. Now, this should work, but it doesn't. I filed a bug, SPL-108269 to track it, but will probably be a while before anyone gets to it given pretty valid workarounds available. This will affect any modular input you're using, not just Stream.
My suggestion to you is to put your configurations you want to monitor in your own app, or at a the very least, put them in $SPLUNKHOME/etc/apps/search/local/inputs.conf and not change the default of having your hostname determined at First Time Run and written to $SPLUNKHOME/etc/system/local/inputs.conf.
Thanks Clint! Will work on tweaking the inputs per the note above