topic Combined logs, props.conf transforms.conf and inputs.conf in Getting Data In

Combined logs, props.conf transforms.conf and inputs.conf

bpgoodm — Wed, 30 Sep 2020 02:30:42 GMT

I have a centralized syslog server which I forward all other server logs to. All of those logs are combined per log type such as /var/log/messages, /var/log/secure, etc. I've setup my transforms.conf file to take the hostname out of each line and assign it as the host per log file line. I've associated the transform to the various logs. After restarting the splunk forwarder I'm still not seeing my logs being broken out by host. I'm wondering if it's because of the host value in inputs.conf and if that's overriding the dynamic host value I'm creating via props and transforms. Here is what I have so far:

etc/apps/search/local/transforms.conf

[hostname]
DEST_KEY = MetaData:Host
REGEX = (?:\S+ ){2}(\S+)
FORMAT = host::$1

etc/system/local/props.conf

[source::.../messages]
TRANSFORMS-bg = hostname
SHOULD_LINEMERGE = False

Example from splunk show config inputs

[monitor:///var/log/messages*]
_rcvbuf=1572864
disabled=false
host=[splunk_forwarder_hostname]
index=[index_name]
sourcetype=[source_type]

My concern is the host= attribute associated with every monitor entry. Is this the source of my issues or do I need to be looking somewhere else?

Bert

Re: Combined logs, props.conf transforms.conf and inputs.conf

gcusello — Thu, 10 Oct 2019 06:45:28 GMT

Hi bpgoodm,
let me understand:

you have a server with Universal Forwarders that receives syslogs,
then this server sends logs to Indexer without intermediate forwarders,

Is it correct?

if this is you situation, the problem is on the location of props.conf and transforms.conf: they must be on Indexers not on Universal Forwarders.

There are only two exceptions to this rule:

when there's an intermediate Heavy Forwarder that they must be here,
if you have to ingest csv or xml or json, in other words indexed extractions, in this case they must be on Universal Forwarders.

Bye.
Giuseppe

Re: Combined logs, props.conf transforms.conf and inputs.conf

bpgoodm — Thu, 10 Oct 2019 12:58:00 GMT

Hi Giuseppe,

Ah, thank you. I didn't realize. Your assumptions are correct. I'm setting this stuff up on the Universal Forwarder and the Universal Forwarder is sending the logs to the Indexer. The exceptions you mention don't apply to this situation. Thank you again for the information.

Bert