topic Re: How to over ride the index for event from certain hosts? in Getting Data In

How to over ride the index for event from certain hosts?

Glasses — Tue, 07 Jul 2020 22:38:16 GMT

Its been awhile since I setup an props/transforms override, but I never had so much trouble.

I have 20 Foo-appliances sending data to a TCP Listener (unique high port) on an HF version 7.3.3.

The inputs.conf for this data is in > /opt/splunk/etc/apps/search/local

[tcp://12345]

connection_host = dns
index = foo
sourcetype = foo_log

I have 2 Foo-appliances that are sending a different format to the same HF TCP port and I want to send those to a different index=bar sourcetype=bar_log.

host 1 = abcd-1234-efgh-blahblah.com

host 2 = zyxw-9876-ghyh-blahblah.com

I have tried a number of combinations of props and transforms with no luck.

The HF does not index the data, just forwards the data to the indexers.

Please advise which directory to create an override stanza in the props and transforms, .../system/local or .../search/local ?

Here is what I tried and had no luck...

Sample props.conf

[source::tcp:12345]

Transforms-OverRide-Foo-index = OverRide-Foo

Transforms-OverRide-Foo-sourcetype = OverRide-Foo_log

Sample transforms.conf

[OverRide-Foo]

REGEX = (abcd* | zyxw* )

DEST_KEY = _MetaData:Index

FORMAT = bar

[OverRide-Foo_Log]

REGEX = (abcd* | zyxw*)

DEST_KEY = MetaData:Sourcetype

FORMAT = bar_log

I looked at a lot of the examples but I must be misunderstanding.

Thanks in advance!

Re: How to over ride the index for event from certain hosts?

richgalloway — Wed, 08 Jul 2020 00:40:46 GMT

If the distinguishing characteristic is the host field then REGEX will not find it as it looks at _raw. Try putting this in props.conf on the HF:

[host::abcd-1234-efgh-blahblah.com] Transforms-OverRide-Foo-index = OverRide-Foo Transforms-OverRide-Foo-sourcetype = OverRide-Foo_log [host::zyxw-9876-ghyh-blahblah.com] Transforms-OverRide-Foo-index = OverRide-Foo Transforms-OverRide-Foo-sourcetype = OverRide-Foo_log

Then transforms.conf becomes

[OverRide-Foo] REGEX = . DEST_KEY = _MetaData:Index FORMAT = bar [OverRide-Foo_Log] REGEX = . DEST_KEY = MetaData:Sourcetype FORMAT = bar_log

Re: How to over ride the index for event from certain hosts?

Glasses — Wed, 08 Jul 2020 13:34:48 GMT

Thank you.

I followed your instructions, but it did not work...

I don't have any conflicts in .../system/local, and I made the change in .../etc/apps/search/local < where the input was created. Should I move it to ...system/local?

I believe your config is correct, but something must be conflicting or the HF does not see the Host =abcd-1234-efgh-blahblah.com

in the _raw log.

I tried changing the host to the IP and still no luck.

Any ideas ?

I will keep looking for conflicts as there is so much old tangled garbage everywhere in this deployment.

Thank you

Re: How to over ride the index for event from certain hosts?

Glasses — Wed, 08 Jul 2020 13:49:14 GMT

is there another way to write the props / transforms to look at the tcp source and regex _raw , for the hostname or IP , like

Props.conf

[source::tcp:12345]
Transforms-OverRide-Foo-index = OverRide-Foo
Transforms-OverRide-Foo-sourcetype = OverRide-Foo_log

transforms.conf
[OverRide-Foo]
REGEX = (abcd* | 1.2.3.4 )
DEST_KEY = _MetaData:Index
FORMAT = bar

[OverRide-Foo_Log]
REGEX = (abcd* | 1.2.3.4 )
DEST_KEY = MetaData:Sourcetype
FORMAT = bar_log

Re: How to over ride the index for event from certain hosts?

richgalloway — Wed, 08 Jul 2020 13:52:46 GMT

There's no need to move the configuration to etc/system/local.

You installed the config files on the HF and then restarted the HF, right?

The value following host:: should be whatever appears in the host field when you search for events from that server.

btool can help you figure out where configurations are being set.

splunk btool --debug inputs list | more

Re: How to over ride the index for event from certain hosts?

Glasses — Wed, 08 Jul 2020 14:11:56 GMT

yes, I always restart after a wq of the conf.

This is weird I don't see anything abnormal with btool.

I am wondering if there is something funky with the logs the HF receives, like its not seeing the host field but I know the the name and/or ip is in the raw event.

Re: How to over ride the index for event from certain hosts?

richgalloway — Wed, 08 Jul 2020 14:14:37 GMT

Try adding this to your transforms.conf stanzas

SOURCE_KEY = metadata:host

Re: How to over ride the index for event from certain hosts?

Glasses — Wed, 08 Jul 2020 14:21:00 GMT

still no luck with

SOURCE_KEY = MetaData:Host

Re: How to over ride the index for event from certain hosts?

Glasses — Tue, 21 Jul 2020 13:27:18 GMT

FYI - here is what worked. Evidently I had a syntax error...

inputs.conf

[tcp://12345]

connection_host = dns
index = foo
sourcetype = foo_log

props.conf

TRANSFORMS-1_extract_hostname = foo_host
TRANSFORMS-2_overide_foo_index = foo_index_override
TRANSFORMS-3_overide_foo_sourcetype = foo_sourcetype_override

transforms.conf

[foo_host]
DEST_KEY = MetaData:Host
FORMAT = host::$1

[foo_index_override]
SOURCE_KEY = MetaData:Host
REGEX = host::abc123.+
DEST_KEY = _MetaData:Index
FORMAT = bar

[foo_sourcetype_override]
SOURCE_KEY = MetaData:Host
REGEX = host::abc123.+
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::bar_logs