Getting Data In

How to over ride the index for event from certain hosts?

Glasses
Builder

Its been awhile since I setup an props/transforms override, but I never had so much trouble.

I have 20 Foo-appliances sending data to a TCP Listener (unique high port) on an HF version 7.3.3.

The inputs.conf for this data is in > /opt/splunk/etc/apps/search/local

[tcp://12345]

connection_host = dns
index = foo
sourcetype = foo_log

I have 2 Foo-appliances that are sending a different format to the same HF TCP port and I want to send those to a different index=bar sourcetype=bar_log.

host 1 = abcd-1234-efgh-blahblah.com

host 2 = zyxw-9876-ghyh-blahblah.com

I have tried a number of combinations of props and transforms with no luck.

The HF does not index the data, just forwards the data to the indexers.

Please advise which directory to create an override stanza in the props and transforms, .../system/local or .../search/local  ?

 

Here is what I tried and had no luck... 

Sample props.conf

[source::tcp:12345]

Transforms-OverRide-Foo-index = OverRide-Foo

Transforms-OverRide-Foo-sourcetype = OverRide-Foo_log

Sample transforms.conf

[OverRide-Foo]

REGEX = (abcd* | zyxw* )  

DEST_KEY = _MetaData:Index

FORMAT = bar

[OverRide-Foo_Log]

REGEX = (abcd* | zyxw*)

DEST_KEY = MetaData:Sourcetype

FORMAT = bar_log

 

I looked at a lot of the examples but I must be misunderstanding.

Thanks in advance!

Labels (2)
0 Karma
1 Solution

Glasses
Builder

FYI - here is what worked. Evidently I had a syntax error...

inputs.conf

[tcp://12345]

connection_host = dns
index = foo
sourcetype = foo_log

props.conf

TRANSFORMS-1_extract_hostname = foo_host
TRANSFORMS-2_overide_foo_index = foo_index_override
TRANSFORMS-3_overide_foo_sourcetype = foo_sourcetype_override

transforms.conf

[foo_host]
DEST_KEY = MetaData:Host
FORMAT = host::$1

[foo_index_override]
SOURCE_KEY = MetaData:Host
REGEX = host::abc123.+
DEST_KEY = _MetaData:Index
FORMAT = bar

[foo_sourcetype_override]
SOURCE_KEY = MetaData:Host
REGEX = host::abc123.+
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::bar_logs   

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

If the distinguishing characteristic is the host field then REGEX will not find it as it looks at _raw.  Try putting this in props.conf on the HF:

[host::abcd-1234-efgh-blahblah.com]
Transforms-OverRide-Foo-index = OverRide-Foo
Transforms-OverRide-Foo-sourcetype = OverRide-Foo_log

[host::zyxw-9876-ghyh-blahblah.com]
Transforms-OverRide-Foo-index = OverRide-Foo
Transforms-OverRide-Foo-sourcetype = OverRide-Foo_log

Then transforms.conf becomes

[OverRide-Foo]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = bar

[OverRide-Foo_Log]
REGEX = .
DEST_KEY = MetaData:Sourcetype
FORMAT = bar_log
---
If this reply helps you, Karma would be appreciated.

Glasses
Builder

FYI - here is what worked. Evidently I had a syntax error...

inputs.conf

[tcp://12345]

connection_host = dns
index = foo
sourcetype = foo_log

props.conf

TRANSFORMS-1_extract_hostname = foo_host
TRANSFORMS-2_overide_foo_index = foo_index_override
TRANSFORMS-3_overide_foo_sourcetype = foo_sourcetype_override

transforms.conf

[foo_host]
DEST_KEY = MetaData:Host
FORMAT = host::$1

[foo_index_override]
SOURCE_KEY = MetaData:Host
REGEX = host::abc123.+
DEST_KEY = _MetaData:Index
FORMAT = bar

[foo_sourcetype_override]
SOURCE_KEY = MetaData:Host
REGEX = host::abc123.+
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::bar_logs   

0 Karma

Glasses
Builder

is there another way to write the props / transforms to look at the tcp source and regex _raw , for the hostname or IP , like 

Props.conf

[source::tcp:12345]
Transforms-OverRide-Foo-index = OverRide-Foo
Transforms-OverRide-Foo-sourcetype = OverRide-Foo_log

transforms.conf
[OverRide-Foo]
REGEX = (abcd* | 1.2.3.4 )
DEST_KEY = _MetaData:Index
FORMAT = bar

[OverRide-Foo_Log]
REGEX = (abcd* | 1.2.3.4 )
DEST_KEY = MetaData:Sourcetype
FORMAT = bar_log

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Try adding this to your transforms.conf stanzas

SOURCE_KEY = metadata:host
---
If this reply helps you, Karma would be appreciated.
0 Karma

Glasses
Builder

still no luck with 

SOURCE_KEY = MetaData:Host
0 Karma

Glasses
Builder

Thank you.

I followed your instructions, but it did not work...

I don't have any conflicts in .../system/local, and I made the change in .../etc/apps/search/local  < where the input was created.  Should I move it to ...system/local?

I believe your config is correct, but something must be conflicting or the HF does not see the Host =abcd-1234-efgh-blahblah.com

in the _raw log. 

I tried changing the host to the IP and still no luck.

Any ideas ?

I will keep looking for conflicts as there is so much old tangled garbage everywhere in this deployment.

Thank you 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

There's no need to move the configuration to etc/system/local.

You installed the config files on the HF and then restarted the HF, right?

The value following host:: should be whatever appears in the host field when you search for events from that server.

btool can help you figure out where configurations are being set.

splunk btool --debug inputs list | more
---
If this reply helps you, Karma would be appreciated.
0 Karma

Glasses
Builder

yes, I always restart after a wq of the conf.

This is weird I don't see anything abnormal with btool.

I am wondering if there is something funky with the logs the HF receives, like its not seeing the host field but I know the the name and/or ip is in the raw event.

 

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...