Getting Data In

Override host field with event data

Engager

Hello,

I am indexing some data from a file monitor and i want to override the host field with data that lays inside the events. Below is a sample of the data and the values i want for the host field with bold.

Mon Oct 09 2017 15:24:18 SE-001 sshd[5905]: Failed password for invalid user postgres from 49.212.64.138 port 4856 ssh2
Mon Oct 09 2017 15:24:13 ACME-005 sshd[2792]: Failed password for nsharpe from 10.2.10.163 port 1148 ssh2
Mon Oct 09 2017 15:24:12 ops-sys-006 sshd[4105]: Failed password for sync from 233.77.49.94 port 4595 ssh2
Mon Oct 09 2017 15:24:19 PROD-MFS-001 sshd[74897]: pam_unix(sshd:session): session closed for user nsharpe by (uid=0)
Mon Oct 09 2017 15:24:07 PROD-MFS-001 su: pam_unix(su:session): session closed for user root

The data is indexed under linux_secure sourcetype. In order to achieve the host overriding, i added one props.conf and one transforms.conf stanza in /etc/system/local on the indexers:

props.conf
[linux_secure]
TRANSFORMS-sethost = set_hostname_linux_secure
SHOULD_LINEMERGE = false

transforms.conf
[set_hostname_linux_secure]
REGEX = (?<=:\d{2}\s).*?(?=\s)
FORMAT = host::$1
DEST_KEY = MetaData:Host

The above configuration is not working, and the events are still indexing with host = the name of the forwarder where they come from.

Any idea what's wrong with this configuration and how can i implement the host overriding?

Thanks a lot!

0 Karma
1 Solution

SplunkTrust
SplunkTrust

To me it looks like your problem is in your config. You are calling out $1 in the FORMAT line, but you don't actually have a capture group that you can use. Try this:

REGEX = (?<=:\d{2}\s)(\S+)(?=\s)

It should at least have something in $1 for it to set the host with (the (\S+) will be the only capture group that returns a value).

View solution in original post

Splunk Employee
Splunk Employee

Hey @raduand, if they solved your problem, remember to "√Accept" an answer to award karma points 🙂

0 Karma

Engager

problem is not solved yet 🙂

0 Karma

SplunkTrust
SplunkTrust

To me it looks like your problem is in your config. You are calling out $1 in the FORMAT line, but you don't actually have a capture group that you can use. Try this:

REGEX = (?<=:\d{2}\s)(\S+)(?=\s)

It should at least have something in $1 for it to set the host with (the (\S+) will be the only capture group that returns a value).

View solution in original post

Engager

You are right about the config problem, but even after updating the Regex expression to capture a group that returns a value the host overriding still doesn't work.

Any other suggestion or idea how to troubleshoot this?

Thank you!

0 Karma

Motivator

Are you using distributed environment???

These configuration should be added on indexer if you are using universal forwarder.

0 Karma

Engager

Yes, i am using distributed environment. The data is coming from a heavy forwarder. The configuration was placed on the indexers and the host overriding was not working.

I just placed props.conf and transforms.conf on the Heavy Forwarder and it's working like a charm.

I need to know why the indexers are not performing this parsing.

0 Karma

Engager

I'm running Splunk 6.6.3

0 Karma

Motivator

If you are using Heavy Forwarder then you have to place these configurations on HF only. Because with HF, Splunk completes parsing on HF itself and indexer only indexes data. Please refer following link to know more about Splunk indexing.
https://wiki.splunk.com/Community:HowIndexingWorks

0 Karma

Engager

Great information, thanks! problem solved then.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!