This seems to be a common question and I've read several previous discussions. The issue seems to be that the default Linux UF config 'knows' the FQDN and returns that for log-flies which do not have a 'host' value, but then some of the most important files, e.g. /var/log/messages, do include a host and so the UF 'defers' to that value, even if it's not FQDN.
The simplest solution has been to update your Linux servers' rsyslog config to record the FQDN to all logs. But I am trying to avoid walking my environment to make that change.
Instead I am looking for a specific example of the required transform.conf, which I could push to all UFs (via a deploy-app) so that they 're-substitute' the FQDN for the short 'host' value. Can someone please show me how? Thank you!
P.S. I am also trying to avoid doing this at the indexer, both because it is unclear if the indexer has access to the FQDN and also because this is a shared environment and I do not have permission to edit this system-wide; I am only trying to fix my dept's servers.
Steps given in page works for both Splunk enterprise and cloud. You can create props.conf and transforms.conf in your forwarder add-on and deploy it on UFs.
Thanks, that gets me closer! The only example in that doc, however, is for using regex to extract the 'host' field from the particular log-file. I'm trying to use the FQDN 'host' stored (I believe) in the local 'server.conf', i.e., the expected solution will not need to mention any particular log-file. So how do I specify that?
Also, we are running the Splunk Add-on for Unix And Linux, so we are pushing that 'SplunkTAnix' deploy-app, which includes the following stanza in 'inputs.conf'
index = os
Is that the stanza I need to reference in my new 'props.conf'? Also, can I create that custom props.conf in the same deploy-app where I specify my own custom inputs.conf?
OK, then you just need add host = $decideOnStartup to monitor stanza in inputs.conf in SplunkTAnix app and push. This sets the host field to the hostname of executing machine on each splunkd startup. No props.conf and transforms.conf is required.
[monitor:///var/log] index = os host = $decideOnStartup
Thank you for the $DecideOnStartup suggestion! Unfortunately, I couldn't make it work, even after a full restart of my test server.
Actually, I even tried hard-coding the FQDN and that still failed, i.e., the references for 'syslog' in the transforms.conf of the SplunkTAnix seems to take precedence (over the inputs.conf)
It's case sensitive you need to set $decideOnStartup not $DecideOnStartup. And also check if host is set in $SPLUNK_UFHOME/etc/system/local/inputs.conf on UFs. Remove the host setting if it exists in system local.
yes, I had copy-and-pasted your example so the case-sensitive part was correct (assuming your example was correct). My initial reply I wrote it by hand, then corrected it.
My default inputs.conf has the FQDN! This is what is being overridden by the SplunkTAnix transforms for syslog-type logs:
host = server123.mycompany.com
Use btool to see host setting set for syslog source and from which config file. And remove host setting from that config file for the sourcetype.
$SPLUNK_UFHOME/bin/splunk cmd btool inputs list --debug
Refer to this wiki article first:
You will see that transforms comes in much later on in the pipeline. It's not a config that will end up on the UF to append the FQDN at the UF, it happens at the Indexer tier.
You can create a lookup table to fix this issue, though this would be ongoing maintenance:
1) Create a search to output all the host in a table
2) Export the host list and create a new column, which has the FQDN entry
3) Import the lookup file onto the SH to an existing / new app
4) Run the lookup search as part of the spl and it will return thee FQDN value
Alternatively (preferred), you can do it to OS way, more work initially though this will help create a consistent process
Thank you for the link to that Wiki article.
Lookup table seems unnecessary since the UF already knows the FQDN, I just need to retrieve it.