My scripted input is working, but why is it assigning an incorrect hostname for 1 of 6 target hosts?

[pipegrep]

I have a simple scripted input that runs a powermt command periodically.

<apps>/powermt/bin/powermt.sh

#!/bin/bash
sudo powermt display

<apps>/powermt/local/inputs.conf

##### Powermount scripted Inputs ######1
[script:///opt/splunkforwarder/etc/apps/powermt/bin/powermt.sh]
## Run every 15 minutes
disabled = false
interval = 15
source = powermnt
sourcetype = script:///opt/splunkforwarder/etc/apps/powermt/bin/powermt.sh]

<apps>/local/props.conf

[powermt]
DATETIME_CONFIG = CURRENT

The app deploys and runs as it should on 5 out of the 6 target hosts otherwise , however on the 6th host it attributes the data to an incorrect host name (a test server that has been retired).

The good news is that there's lots documentation about this. The bad news is that there's lots of documentation about this. While I go back and convert several hours of my day into unraveling the root issue, I'm wondering if there's a way I can force the script to force/set the local hostname. This would relieve quite a bit of my immediate pressure.

[acharlieh · Accepted solution]

In inputs.conf there is a host attribute that you could set on the script stanza to seed the appropriate value (and assuming your indexer isn't overriding this value at parse time, this is what your events will be set to). Let's take a peek at the resolved config, so in your splunk home, run: bin/splunk btool inputs list script --debug

This will spit out the resolved configuration of all your script stanzas (for a global time configuration as inputs is), and the path to the inputs.conf file that each line is resolved from. You may see a host already being set for the stanza in question (likely in etc/system/local/inputs.conf ).

If you're curious about this resolution of files you may be interested in docs about how this resolution works or even more documentation about btool, or a video on how to “Support Splunk like A Support Professional – Episode 3 – Troubleshooting Configuration Issues”.

Now unrelated to the host problem, you have one other issue, that your props.conf is configuring a stanza to point to the sourcetype of powermt, but your stanza is setting the source to powermt.

[pipegrep]

Thank you. Yes, btool is my friend.

Sure enough;
/opt/splunkforwarder/etc/system/local/inputs.conf had a server name forced into it. I have no idea why, though that probably merits some review. In the meantime I'll clear out the old data and let this start reindexing as it should.

[acharlieh]

It's been my observation that, Splunk (currently, at least since 6.0) sets a host name into that file on first run. What I suspect happened, is that the Splunk UF was setup once on one box, and then the directory was cloned between machines or a source machine was imaged with it. I think there's instructions for steps to follow to install a UF as part of a system image somewhere, but I'd have to go digging for a link.

[pipegrep]

This was a physical system that had been repurposed to a degree.I bet you're right.