topic Re: Override host field with event data in Getting Data In

Override host field with event data

raduand — Tue, 29 Sep 2020 16:12:43 GMT

Hello,

I am indexing some data from a file monitor and i want to override the host field with data that lays inside the events. Below is a sample of the data and the values i want for the host field with bold.

Mon Oct 09 2017 15:24:18 SE-001 sshd[5905]: Failed password for invalid user postgres from 49.212.64.138 port 4856 ssh2
Mon Oct 09 2017 15:24:13 ACME-005 sshd[2792]: Failed password for nsharpe from 10.2.10.163 port 1148 ssh2
Mon Oct 09 2017 15:24:12 ops-sys-006 sshd[4105]: Failed password for sync from 233.77.49.94 port 4595 ssh2
Mon Oct 09 2017 15:24:19 PROD-MFS-001 sshd[74897]: pam_unix(sshd:session): session closed for user nsharpe by (uid=0)
Mon Oct 09 2017 15:24:07 PROD-MFS-001 su: pam_unix(su:session): session closed for user root

The data is indexed under linux_secure sourcetype. In order to achieve the host overriding, i added one props.conf and one transforms.conf stanza in /etc/system/local on the indexers:

props.conf
[linux_secure]
TRANSFORMS-sethost = set_hostname_linux_secure
SHOULD_LINEMERGE = false

transforms.conf
[set_hostname_linux_secure]
REGEX = (?<=:\d{2}\s).*?(?=\s)
FORMAT = host::$1
DEST_KEY = MetaData:Host

The above configuration is not working, and the events are still indexing with host = the name of the forwarder where they come from.

Any idea what's wrong with this configuration and how can i implement the host overriding?

Thanks a lot!

Re: Override host field with event data

cpetterborg — Mon, 09 Oct 2017 18:30:24 GMT

To me it looks like your problem is in your config. You are calling out $1 in the FORMAT line, but you don't actually have a capture group that you can use. Try this:

REGEX = (?<=:\d{2}\s)(\S+)(?=\s)

It should at least have something in $1 for it to set the host with (the (\S+) will be the only capture group that returns a value).

Re: Override host field with event data

lfedak_splunk — Tue, 10 Oct 2017 00:17:55 GMT

Hey @raduand, if they solved your problem, remember to "√Accept" an answer to award karma points 🙂

Re: Override host field with event data

raduand — Tue, 10 Oct 2017 08:28:31 GMT

You are right about the config problem, but even after updating the Regex expression to capture a group that returns a value the host overriding still doesn't work.

Any other suggestion or idea how to troubleshoot this?

Thank you!

Re: Override host field with event data

raduand — Tue, 10 Oct 2017 11:04:13 GMT

problem is not solved yet 🙂

Re: Override host field with event data

hardikJsheth — Tue, 10 Oct 2017 13:05:06 GMT

Are you using distributed environment???

These configuration should be added on indexer if you are using universal forwarder.

Re: Override host field with event data

raduand — Tue, 10 Oct 2017 13:09:15 GMT

Yes, i am using distributed environment. The data is coming from a heavy forwarder. The configuration was placed on the indexers and the host overriding was not working.

I just placed props.conf and transforms.conf on the Heavy Forwarder and it's working like a charm.

I need to know why the indexers are not performing this parsing.

Re: Override host field with event data

raduand — Tue, 10 Oct 2017 13:11:47 GMT

I'm running Splunk 6.6.3

Re: Override host field with event data

hardikJsheth — Tue, 10 Oct 2017 13:15:55 GMT

If you are using Heavy Forwarder then you have to place these configurations on HF only. Because with HF, Splunk completes parsing on HF itself and indexer only indexes data. Please refer following link to know more about Splunk indexing.
https://wiki.splunk.com/Community:HowIndexingWorks

Re: Override host field with event data

raduand — Tue, 10 Oct 2017 13:19:38 GMT

Great information, thanks! problem solved then.