Getting Data In

Why are multiple host names being reported for the same host?

Michael
Contributor

'Morning...
I have a v6.5, clustered environment (deployment server), Universal Forwarder on all hosts.

I am getting several Linux systems reporting in with two names, shortname and FQDN. But not all of them are doing this, even members of the same Server Class.

It seems that all the shortnames are only pulling a sourcetype of syslog or linux_messages_syslog and are only source=/var/log/messages.

The FQDNs are showing appropriate sourcetypes and sources (all under /var/log/ -- but NOT messages).

I have a very simple inputs.conf being deployed:

[monitor:///var/log]
index = servers
disabled = 0

I confirmed that syslog is not configured on these to also send to my heavy forwarders. They are reporting in to the Forwarder Management interface as one system (mixture of short and FQDN).

I haven't found a lot of mentions of this here -- I guess this is not very common...?

Thoughts?
Thanks!
Michael

ridwanahmed
Path Finder

For someone else with this issue-- the sourcetype of "syslog" has a specific transform set up to pull out the hostname from the logs. It doesn't use the inputs.conf or server.conf name

See this answer:
https://answers.splunk.com/answers/55751/host-field-getting-overwritten-in-syslog-processing.html

DalJeanis
SplunkTrust
SplunkTrust

Are some of the logs duplicated, or is it either/or?

If either/or, then can you post sanitized versions of each kind?

0 Karma

Michael
Contributor

None of them are duplicated.

Of the three specific ones (in this section of my organization) that I'm narrowing down on, they send everything fine from /var/log/ using their FQDN -- but only the /var/log/messages file is reported using the short-name. They have other Linux (RHEL and CentOs) in that area that are reporting in using shortnames only. I'm trying to find out how they're different...

Ah, I see why you wanted a sample of the logs -- the /var/log/message file does include the hostname (short) -- seems that Splunk is pulling the name from there.(?) In the other log files, it does not included a name -- so it's getting it from DNS (hence, the FQDN).

i.e.:
Mar 13 06:18:29 servername dhclient[2958]: DHCPACK from 10.14.8.82 (xid=0x369db4ff)
v.s.
type=CRED_DISP msg=audit(1489419002.019:169929): user pid=26065 uid=0 auid=0 ses=25846 msg='op=PAM:setcred acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'

Is that where you were going with that? I still see other hosts in the same area that are reporting with the same syntax -- but they're not producing duplicate names.

OK, now, how to fix that?

(I love it when I'm apparently the only one "out there" that's experienced are particular issue... 😉

Cheers,

0 Karma

ridwanahmed
Path Finder

Did you ever find a solution to this? I'm having a similar issue-- only source=var/log/messages goes to host=hostname, while the other logs have host=fqdn

my inputs.conf--the only place where this host name is defined (it's not in server.conf)

[default]
host = myhost.mydomain.com
[monitor:///var/log/messages]
disabled = false
sourcetype = syslog
index = myindex_1

[monitor:///usr/local/tomcat/logs/logname.log]
disabled = false
sourcetype = log4j i
index = myindex_2

Thanks for any comments.

0 Karma

Michael
Contributor

FYI, trying $decideonStartUp didn't seem to work. To recap:

[monitor:///var/log]
host = $decideOnStartup
index = atl
disabled = 0

Mix of FQDN and short names sending in these:
/var/log/audit/audit.log
/var/log/cron

/var/log/mcelog
/var/log/up2date
/var/log/rhsm/rhsm.log
etc...

short only no FQDN
/var/log/messages

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi Michael,
If you have only forwarders and none syslog you have to verify the servername associated to the Splunk Forwarder.
You can verify it in your servers in $SPLUNK_HOME/etc/system/local/server.conf and in $SPLUNK_HOME/etc/system/local/inputs.conf.
Servername is associated at the installation time from the server hostname.
If you want, you can modify it but in both the conf files.
Bye.
Giuseppe

0 Karma

Michael
Contributor

Yes, I know where to find the hostname, but thanks.

For what it's worth, I've confirmed that both the inputs.conf and server.conf have the FQDN of the system.

Even /etc/hostname on the system has the FQDN.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...