topic Re: After installing a Splunk 6.4 universal forwarder, why are events indexed with the shortname instead of FQDN for the hostname? in Getting Data In

After installing a Splunk 6.4 universal forwarder, why are events indexed with the shortname instead of FQDN for the hostname?

lib_systems — Tue, 29 Sep 2020 09:30:15 GMT

After an initial installation of the Universal Forwarder (6.4.0), I immediately changed the hostname values to use the FQDN:

./splunk set servername myserver.domain.com
./splunk set default-hostname myserver.domain.com

I then restart the Universal Forwarder service and confirm the changes in the following conf files:

/opt/splunkforwarder/etc/system/local/inputs.conf:

[default]
host = myserver.domain.com

/opt/splunkforwarder/etc/system/local/server.conf:

[general]
serverName = myserver.domain.com
...

However, unless I explicitly specify the FQDN hostname, when I add a new monitor (sourcetypes linux_secure and linux_messages_syslog), the events are indexed with the shortname.

The splunkd.log seems to suggest it is not honoring the default hostname I set for the inputs.conf (oddly, the servername in server.conf seems to stick):

...
04-18-2016 15:10:37.451 -0400 INFO  ServerConfig - My server name is "myserver.domain.com".
04-18-2016 15:10:37.452 -0400 INFO  ServerConfig - Found no site defined in server.conf
04-18-2016 15:10:37.452 -0400 INFO  ServerConfig - My hostname is "myserver".
...

This behavior is reproducible on multiple hosts. Is there something else I'm missing? Any advice is appreciated. Thanks.

Re: After installing a Splunk 6.4 universal forwarder, why are events indexed with the shortname instead of FQDN for the hostname?

aladda_splunk — Wed, 20 Apr 2016 19:15:34 GMT

What do you get when you run a btool list on inputs? This command from you $SPLUNK_HOME/bin directory
./splunk cmd btool inputs list --debug | grep host

Re: After installing a Splunk 6.4 universal forwarder, why are events indexed with the shortname instead of FQDN for the hostname?

lib_systems — Tue, 29 Sep 2020 09:27:44 GMT

All results from this command are using the desired FQDN:
/opt/splunkforwarder/etc/system/local/inputs.conf host = myserver.domain.com
/opt/splunkforwarder/etc/system/local/inputs.conf host = myserver.domain.com
/opt/splunkforwarder/etc/system/local/inputs.conf host = myserver.domain.com
/opt/splunkforwarder/etc/system/local/inputs.conf host = myserver.domain.com
/opt/splunkforwarder/etc/system/local/inputs.conf host = myserver.domain.com
/opt/splunkforwarder/etc/system/local/inputs.conf host = myserver.domain.com
/opt/splunkforwarder/etc/system/local/inputs.conf host = myserver.domain.com
/opt/splunkforwarder/etc/system/local/inputs.conf host = myserver.domain.com
/opt/splunkforwarder/etc/system/local/inputs.conf host = myserver.domain.com
/opt/splunkforwarder/etc/system/local/inputs.conf host = myserver.domain.com
/opt/splunkforwarder/etc/system/local/inputs.conf host = myserver.domain.com
/opt/splunkforwarder/etc/system/local/inputs.conf host = myserver.domain.com
/opt/splunkforwarder/etc/system/default/inputs.conf connection_host = ip
/opt/splunkforwarder/etc/system/local/inputs.conf host = myserver.domain.com
/opt/splunkforwarder/etc/system/default/inputs.conf connection_host = dns
/opt/splunkforwarder/etc/system/local/inputs.conf host = myserver.domain.com
/opt/splunkforwarder/etc/system/default/inputs.conf connection_host = ip
/opt/splunkforwarder/etc/system/local/inputs.conf host = myserver.domain.com

I am seeing other posts describing the same problem [1] [2]. Both allude to the fact that regardless of the settings I configure, the host field is being overridden by the default transformers for the linux_messages_syslog sourcetype. I've tried the suggestions of creating my own custom props.conf in the /local directory to override the default transformers, however, that has not worked for me (I did this on the universal forwarder though it's still not clear to me if that should be done there or on the indexer).

Those suggestions aside, what still puzzles me is that the splunkd.log indicates the universal forwarder is not honoring my settings immediately upon startup and uses the incorrect hostname. It feels like fiddling with custom props.conf isn't going to help in this case.

[1] https://answers.splunk.com/answers/149755/universal-forwarder-6-1-2-hostname-not-equal-to-servername-inputs-conf.html
[2] https://answers.splunk.com/answers/6895/can-i-prevent-the-default-index-time-extraction-for-the-host-field-to-occur-for-events-of-the-syslog-sourcetype.html

Re: After installing a Splunk 6.4 universal forwarder, why are events indexed with the shortname instead of FQDN for the hostname?

dolivasoh — Thu, 21 Apr 2016 04:53:12 GMT

I've noticed that 6.4 is much more strict about the serverName key in server.conf. I believe this is the new end-all be-all forwarder name. This became obvious to me in some automation testing using puppet to apply a forwarder upgrade from 6.3.3.

Funny thing is this doesn't appear to be documented in http://docs.splunk.com/Documentation/Splunk/6.4.0/Installation/Aboutupgradingto6.4READTHISFIRST

Re: After installing a Splunk 6.4 universal forwarder, why are events indexed with the shortname instead of FQDN for the hostname?

lib_systems — Thu, 21 Apr 2016 12:26:55 GMT

My server.conf does have the serverName key set to my FQDN and according to splunkd.log it is honoring this setting. The default-hostname remains the problematic piece.

Re: After installing a Splunk 6.4 universal forwarder, why are events indexed with the shortname instead of FQDN for the hostname?

Raschko — Tue, 29 Sep 2020 12:34:04 GMT

The problem is the default parsing of the host field on the indexer for pretrained sourcetype linux_messages_syslog.
You can override it with a local props.conf on the receiving indexer.

Have a look at my other answer here:
https://answers.splunk.com/answers/494084/linux-message-syslog-host-name-issue.html