topic Re: edit Linux agents to always return 'host' as FQDN? in Splunk Search

edit Linux agents to always return 'host' as FQDN?

ttovarzoll — Wed, 04 Mar 2020 22:45:30 GMT

This seems to be a common question and I've read several previous discussions. The issue seems to be that the default Linux UF config 'knows' the FQDN and returns that for log-flies which do not have a 'host' value, but then some of the most important files, e.g. /var/log/messages, do include a host and so the UF 'defers' to that value, even if it's not FQDN.

The simplest solution has been to update your Linux servers' rsyslog config to record the FQDN to all logs. But I am trying to avoid walking my environment to make that change.

Instead I am looking for a specific example of the required transform.conf, which I could push to all UFs (via a deploy-app) so that they 're-substitute' the FQDN for the short 'host' value. Can someone please show me how? Thank you!

P.S. I am also trying to avoid doing this at the indexer, both because it is unclear if the indexer has access to the FQDN and also because this is a shared environment and I do not have permission to edit this system-wide; I am only trying to fix my dept's servers.

Re: edit Linux agents to always return 'host' as FQDN?

manjunathmeti — Wed, 04 Mar 2020 23:23:04 GMT

Hi @ttovarzoll,

Check this page: https://docs.splunk.com/Documentation/SplunkCloud/8.0.2001/Data/Overridedefaulthostassignments

Steps given in page works for both Splunk enterprise and cloud. You can create props.conf and transforms.conf in your forwarder add-on and deploy it on UFs.

Re: edit Linux agents to always return 'host' as FQDN?

anmolpatel — Wed, 04 Mar 2020 23:34:06 GMT

Refer to this wiki article first:
https://wiki.splunk.com/Community:HowIndexingWorks

You will see that transforms comes in much later on in the pipeline. It's not a config that will end up on the UF to append the FQDN at the UF, it happens at the Indexer tier.

You can create a lookup table to fix this issue, though this would be ongoing maintenance:
1) Create a search to output all the host in a table
2) Export the host list and create a new column, which has the FQDN entry
3) Import the lookup file onto the SH to an existing / new app
4) Run the lookup search as part of the spl and it will return thee FQDN value

Alternatively (preferred), you can do it to OS way, more work initially though this will help create a consistent process
https://answers.splunk.com/answers/45899/how-can-i-use-the-fully-qualified-domain-name-fqdn-as-the-host-value-for-var-log-messages.html

Re: edit Linux agents to always return 'host' as FQDN?

ttovarzoll — Wed, 30 Sep 2020 04:28:15 GMT

Thanks, that gets me closer! The only example in that doc, however, is for using regex to extract the 'host' field from the particular log-file. I'm trying to use the FQDN 'host' stored (I believe) in the local 'server.conf', i.e., the expected solution will not need to mention any particular log-file. So how do I specify that?

Also, we are running the Splunk Add-on for Unix And Linux, so we are pushing that 'Splunk_TA_nix' deploy-app, which includes the following stanza in 'inputs.conf'

[monitor:///var/log]
index = os

Is that the stanza I need to reference in my new 'props.conf'? Also, can I create that custom props.conf in the same deploy-app where I specify my own custom inputs.conf?

Re: edit Linux agents to always return 'host' as FQDN?

ttovarzoll — Wed, 04 Mar 2020 23:48:49 GMT

Thank you for the link to that Wiki article.

Lookup table seems unnecessary since the UF already knows the FQDN, I just need to retrieve it.

Re: edit Linux agents to always return 'host' as FQDN?

manjunathmeti — Wed, 30 Sep 2020 04:29:36 GMT

OK, then you just need add host = $decideOnStartup to monitor stanza in inputs.conf in Splunk_TA_nix app and push. This sets the host field to the hostname of executing machine on each splunkd startup. No props.conf and transforms.conf is required.

[monitor:///var/log]
index = os
host = $decideOnStartup

Re: edit Linux agents to always return 'host' as FQDN?

ttovarzoll — Wed, 30 Sep 2020 04:28:46 GMT

Thank you for the $DecideOnStartup suggestion! Unfortunately, I couldn't make it work, even after a full restart of my test server.

Actually, I even tried hard-coding the FQDN and that still failed, i.e., the references for 'syslog' in the transforms.conf of the Splunk_TA_nix seems to take precedence (over the inputs.conf)

Re: edit Linux agents to always return 'host' as FQDN?

manjunathmeti — Thu, 05 Mar 2020 18:33:57 GMT

It's case sensitive you need to set $decideOnStartup not $DecideOnStartup. And also check if host is set in $SPLUNK_UFHOME/etc/system/local/inputs.conf on UFs. Remove the host setting if it exists in system local.

Re: edit Linux agents to always return 'host' as FQDN?

ttovarzoll — Wed, 30 Sep 2020 04:28:49 GMT

yes, I had copy-and-pasted your example so the case-sensitive part was correct (assuming your example was correct). My initial reply I wrote it by hand, then corrected it.

My default inputs.conf has the FQDN! This is what is being overridden by the Splunk_TA_nix transforms for syslog-type logs:

cat /opt/splunkforwarder/etc/system/local/inputs.conf

[default]
host = server123.mycompany.com

Re: edit Linux agents to always return 'host' as FQDN?

harsmarvania57 — Thu, 05 Mar 2020 21:18:57 GMT

Hi,

In splunk installation there is default props configuration for /var/log/messages which assigns sourcetype syslog

[source::.../messages(.\d+)?]
sourcetype = syslog

If you look at syslog sourcetype configuration on your indexer you will able to see below config

props.conf

[syslog]
pulldown_type = true 
maxDist = 3
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 32
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions
SHOULD_LINEMERGE = False
category = Operating System
description = Output produced by many syslog daemons, as described in RFC3164 by the IETF

In above configuration TRANSFORMS = syslog-host is using below REGX which extracts hostname from actual rawdata

transforms.conf

[syslog-host]
DEST_KEY = MetaData:Host
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(\w[\w\.\-]{2,})\]?\s
FORMAT = host::$1

If you are only using syslog sourcetype in your environment for /var/log/messages and not for any other sources and you want to use Linux FQDN (same as given in inputs.conf on UF) for syslog sourcetype then you can try below configuration to override default configuration on Indexer or Heavy Forwarder whichever come first from UF.

props.conf

[syslog]
TRANSFORMS =

Re: edit Linux agents to always return 'host' as FQDN?

ttovarzoll — Thu, 05 Mar 2020 23:29:16 GMT

Thank you for the clear and detailed explanation! It sounds like -- regardless of what I try to do on the agent side -- the Indexer is going to 'transform' the default host value to whatever the regex finds in the log-file.

So I need to either:

edit the configuration of the Indexer (which will affect all depts, something I had hoped to avoid), or
edit all my agents to record the FQDN to all syslog-type logs

In your example of editing the Indexer 'props.conf' are you saying I need to delete all entries for [syslog] or only the 'TRANSFORMS=' setting?

Re: edit Linux agents to always return 'host' as FQDN?

manjunathmeti — Fri, 06 Mar 2020 08:10:20 GMT

Use btool to see host setting set for syslog source and from which config file. And remove host setting from that config file for the sourcetype.

$SPLUNK_UFHOME/bin/splunk cmd btool inputs list --debug

Re: edit Linux agents to always return 'host' as FQDN?

harsmarvania57 — Fri, 06 Mar 2020 16:52:09 GMT

No, you just need to put below config in your Indexer, I will suggest you to test first in your test environment and then move to production.

props.conf

[syslog]
TRANSFORMS =

Re: edit Linux agents to always return 'host' as FQDN?

ttovarzoll — Thu, 26 Mar 2020 20:48:57 GMT

SOLUTION

I decided to 'bite the bullet' and update the local rsyslogd on all our servers. The reason (which I found in another discussion) is that syslog is designed to also act as a 'collector' from other servers, i.e., you cannot assume that all the message in /var/log/messages are local.

All of these suggestions -- and in fact the premise of my original request -- try to ignore the contents of the syslog files in favor of using the FQDN of the reporting-server. But since that initial assumption is invalid, the only safe method is to write the FQDN to the logs, i.e., implement FQDN at the source.

Thank you to everyone who contributed suggestions for fixing this in a centralized manner!