Why are multiple host names being reported for the same host?

[Michael]

'Morning...
I have a v6.5, clustered environment (deployment server), Universal Forwarder on all hosts.

I am getting several Linux systems reporting in with two names, shortname and FQDN. But not all of them are doing this, even members of the same Server Class.

It seems that all the shortnames are only pulling a sourcetype of syslog or linux_messages_syslog and are only source=/var/log/messages.

The FQDNs are showing appropriate sourcetypes and sources (all under /var/log/ -- but NOT messages).

I have a very simple inputs.conf being deployed:

[monitor:///var/log]
index = servers
disabled = 0

I confirmed that syslog is not configured on these to also send to my heavy forwarders. They are reporting in to the Forwarder Management interface as one system (mixture of short and FQDN).

I haven't found a lot of mentions of this here -- I guess this is not very common...?

Thoughts?
Thanks!
Michael

[ridwanahmed]

For someone else with this issue-- the sourcetype of "syslog" has a specific transform set up to pull out the hostname from the logs. It doesn't use the inputs.conf or server.conf name

See this answer:
https://answers.splunk.com/answers/55751/host-field-getting-overwritten-in-syslog-processing.html

[DalJeanis]

Are some of the logs duplicated, or is it either/or?

If either/or, then can you post sanitized versions of each kind?

[Michael]

None of them are duplicated.

Of the three specific ones (in this section of my organization) that I'm narrowing down on, they send everything fine from /var/log/ using their FQDN -- but only the /var/log/messages file is reported using the short-name. They have other Linux (RHEL and CentOs) in that area that are reporting in using shortnames only. I'm trying to find out how they're different...

Ah, I see why you wanted a sample of the logs -- the /var/log/message file does include the hostname (short) -- seems that Splunk is pulling the name from there.(?) In the other log files, it does not included a name -- so it's getting it from DNS (hence, the FQDN).

i.e.:
Mar 13 06:18:29 servername dhclient[2958]: DHCPACK from 10.14.8.82 (xid=0x369db4ff)
v.s.
type=CRED_DISP msg=audit(1489419002.019:169929): user pid=26065 uid=0 auid=0 ses=25846 msg='op=PAM:setcred acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'

Is that where you were going with that? I still see other hosts in the same area that are reporting with the same syntax -- but they're not producing duplicate names.

OK, now, how to fix that?

(I love it when I'm apparently the only one "out there" that's experienced are particular issue... 😉

Cheers,

[ridwanahmed]

Did you ever find a solution to this? I'm having a similar issue-- only source=var/log/messages goes to host=hostname, while the other logs have host=fqdn

my inputs.conf--the only place where this host name is defined (it's not in server.conf)

[default]
host = myhost.mydomain.com
[monitor:///var/log/messages]
disabled = false
sourcetype = syslog
index = myindex_1

[monitor:///usr/local/tomcat/logs/logname.log]
disabled = false
sourcetype = log4j i
index = myindex_2

Thanks for any comments.

[Michael]

FYI, trying $decideonStartUp didn't seem to work. To recap:

[monitor:///var/log]
host = $decideOnStartup
index = atl
disabled = 0

Mix of FQDN and short names sending in these:
/var/log/audit/audit.log
/var/log/cron

/var/log/mcelog
/var/log/up2date
/var/log/rhsm/rhsm.log
etc...

short only no FQDN
/var/log/messages

[gcusello]

Hi Michael,
If you have only forwarders and none syslog you have to verify the servername associated to the Splunk Forwarder.
You can verify it in your servers in $SPLUNK_HOME/etc/system/local/server.conf and in $SPLUNK_HOME/etc/system/local/inputs.conf.
Servername is associated at the installation time from the server hostname.
If you want, you can modify it but in both the conf files.
Bye.
Giuseppe

[Michael]

Yes, I know where to find the hostname, but thanks.

For what it's worth, I've confirmed that both the inputs.conf and server.conf have the FQDN of the system.

Even /etc/hostname on the system has the FQDN.