All Apps and Add-ons

Why is the Splunk App for Stream input tagging host as $decideOnStartup while monitor inputs resolve correctly?

aladda_splunk
Splunk Employee
Splunk Employee

Running into a strange scenario with host field resolution in combination with Splunk App for Stream. All the forwarders are pushed to end-points using chef and they currently copy over the inputs.conf from /etc/system/default into /etc/system/local adding a couple of monitor stanzas to the one in local to monitor additional files. This carries over the default stanza with host=$decideOnStartup into the inputs.conf under local as well

[default]
index         = default
_rcvbuf        = 1572864
host = $decideOnStartup

The monitor inputs however, do get the resolved (correct) host information, as per below. Once the Splunk App for Stream is pushed to the forwarders, the stream data though picks up the host as $decideOnStartup unless either the host is explicitly set in the inputs.conf under /etc/system/local or the forwarder is cleaned up, and no inputs.conf from /etc/system/default is copied over to /etc/system/local which then effectively means the forwarder creates one at startup with the resolved hostname under which then keeps things humming along.

Question is, how are the monitor inputs able to resolve the host=$decideOnStartup to the correct host name, even when placed under /etc/system/local while the Stream input is not? Anyone seen this before and have suggestions on how to get past it, without having to either (a) manually set the host field or (b) redeploy the forwarders without a copy of the inputs.conf from /etc/system/default

Tags (2)
0 Karma
1 Solution

csharp_splunk
Splunk Employee
Splunk Employee

https://answers.splunk.com/answers/3184/hostname-in-inputs-conf.html

By default, Splunk creates $SPLUNK_HOME/etc/system/local/inputs.conf on first time run. You're overwriting that by copying $SPLUNK_HOME/etc/system/default/inputs.conf to $SPLUNK_HOME/etc/system/local/inputs.conf. Now, this should work, but it doesn't. I filed a bug, SPL-108269 to track it, but will probably be a while before anyone gets to it given pretty valid workarounds available. This will affect any modular input you're using, not just Stream.

My suggestion to you is to put your configurations you want to monitor in your own app, or at a the very least, put them in $SPLUNK_HOME/etc/apps/search/local/inputs.conf and not change the default of having your hostname determined at First Time Run and written to $SPLUNK_HOME/etc/system/local/inputs.conf.

View solution in original post

csharp_splunk
Splunk Employee
Splunk Employee

https://answers.splunk.com/answers/3184/hostname-in-inputs-conf.html

By default, Splunk creates $SPLUNK_HOME/etc/system/local/inputs.conf on first time run. You're overwriting that by copying $SPLUNK_HOME/etc/system/default/inputs.conf to $SPLUNK_HOME/etc/system/local/inputs.conf. Now, this should work, but it doesn't. I filed a bug, SPL-108269 to track it, but will probably be a while before anyone gets to it given pretty valid workarounds available. This will affect any modular input you're using, not just Stream.

My suggestion to you is to put your configurations you want to monitor in your own app, or at a the very least, put them in $SPLUNK_HOME/etc/apps/search/local/inputs.conf and not change the default of having your hostname determined at First Time Run and written to $SPLUNK_HOME/etc/system/local/inputs.conf.

aladda_splunk
Splunk Employee
Splunk Employee

Thanks Clint! Will work on tweaking the inputs per the note above

0 Karma

csharp_splunk
Splunk Employee
Splunk Employee

What are you modifying from /etc/system/default/inputs.conf and why are you copying the file wholesale rather than specifying only the parameters you want to override?

0 Karma

aladda_splunk
Splunk Employee
Splunk Employee

Adding a monitor stanza to monitor some additional files. This is an inherited system where the current set-up pushes out a copy of the /etc/system/default/inputs.conf under /etc/system/local. I understand that this isn't the best practice, but looking to see how to fix this problem short of having to blow away the whole UF and reinstall or having to hardcode the host

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...