All Apps and Add-ons

Why is the Splunk App for Stream input tagging host as $decideOnStartup while monitor inputs resolve correctly?

aladda_splunk
Splunk Employee
Splunk Employee

Running into a strange scenario with host field resolution in combination with Splunk App for Stream. All the forwarders are pushed to end-points using chef and they currently copy over the inputs.conf from /etc/system/default into /etc/system/local adding a couple of monitor stanzas to the one in local to monitor additional files. This carries over the default stanza with host=$decideOnStartup into the inputs.conf under local as well

[default]
index         = default
_rcvbuf        = 1572864
host = $decideOnStartup

The monitor inputs however, do get the resolved (correct) host information, as per below. Once the Splunk App for Stream is pushed to the forwarders, the stream data though picks up the host as $decideOnStartup unless either the host is explicitly set in the inputs.conf under /etc/system/local or the forwarder is cleaned up, and no inputs.conf from /etc/system/default is copied over to /etc/system/local which then effectively means the forwarder creates one at startup with the resolved hostname under which then keeps things humming along.

Question is, how are the monitor inputs able to resolve the host=$decideOnStartup to the correct host name, even when placed under /etc/system/local while the Stream input is not? Anyone seen this before and have suggestions on how to get past it, without having to either (a) manually set the host field or (b) redeploy the forwarders without a copy of the inputs.conf from /etc/system/default

Tags (2)
0 Karma
1 Solution

csharp_splunk
Splunk Employee
Splunk Employee

https://answers.splunk.com/answers/3184/hostname-in-inputs-conf.html

By default, Splunk creates $SPLUNK_HOME/etc/system/local/inputs.conf on first time run. You're overwriting that by copying $SPLUNK_HOME/etc/system/default/inputs.conf to $SPLUNK_HOME/etc/system/local/inputs.conf. Now, this should work, but it doesn't. I filed a bug, SPL-108269 to track it, but will probably be a while before anyone gets to it given pretty valid workarounds available. This will affect any modular input you're using, not just Stream.

My suggestion to you is to put your configurations you want to monitor in your own app, or at a the very least, put them in $SPLUNK_HOME/etc/apps/search/local/inputs.conf and not change the default of having your hostname determined at First Time Run and written to $SPLUNK_HOME/etc/system/local/inputs.conf.

View solution in original post

csharp_splunk
Splunk Employee
Splunk Employee

https://answers.splunk.com/answers/3184/hostname-in-inputs-conf.html

By default, Splunk creates $SPLUNK_HOME/etc/system/local/inputs.conf on first time run. You're overwriting that by copying $SPLUNK_HOME/etc/system/default/inputs.conf to $SPLUNK_HOME/etc/system/local/inputs.conf. Now, this should work, but it doesn't. I filed a bug, SPL-108269 to track it, but will probably be a while before anyone gets to it given pretty valid workarounds available. This will affect any modular input you're using, not just Stream.

My suggestion to you is to put your configurations you want to monitor in your own app, or at a the very least, put them in $SPLUNK_HOME/etc/apps/search/local/inputs.conf and not change the default of having your hostname determined at First Time Run and written to $SPLUNK_HOME/etc/system/local/inputs.conf.

aladda_splunk
Splunk Employee
Splunk Employee

Thanks Clint! Will work on tweaking the inputs per the note above

0 Karma

csharp_splunk
Splunk Employee
Splunk Employee

What are you modifying from /etc/system/default/inputs.conf and why are you copying the file wholesale rather than specifying only the parameters you want to override?

0 Karma

aladda_splunk
Splunk Employee
Splunk Employee

Adding a monitor stanza to monitor some additional files. This is an inherited system where the current set-up pushes out a copy of the /etc/system/default/inputs.conf under /etc/system/local. I understand that this isn't the best practice, but looking to see how to fix this problem short of having to blow away the whole UF and reinstall or having to hardcode the host

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...