Its been awhile since I setup an props/transforms override, but I never had so much trouble.
I have 20 Foo-appliances sending data to a TCP Listener (unique high port) on an HF version 7.3.3.
The inputs.conf for this data is in > /opt/splunk/etc/apps/search/local
[tcp://12345]
connection_host = dns
index = foo
sourcetype = foo_log
I have 2 Foo-appliances that are sending a different format to the same HF TCP port and I want to send those to a different index=bar sourcetype=bar_log.
host 1 = abcd-1234-efgh-blahblah.com
host 2 = zyxw-9876-ghyh-blahblah.com
I have tried a number of combinations of props and transforms with no luck.
The HF does not index the data, just forwards the data to the indexers.
Please advise which directory to create an override stanza in the props and transforms, .../system/local or .../search/local ?
Here is what I tried and had no luck...
Sample props.conf
[source::tcp:12345]
Transforms-OverRide-Foo-index = OverRide-Foo
Transforms-OverRide-Foo-sourcetype = OverRide-Foo_log
Sample transforms.conf
[OverRide-Foo]
REGEX = (abcd* | zyxw* )
DEST_KEY = _MetaData:Index
FORMAT = bar
[OverRide-Foo_Log]
REGEX = (abcd* | zyxw*)
DEST_KEY = MetaData:Sourcetype
FORMAT = bar_log
I looked at a lot of the examples but I must be misunderstanding.
Thanks in advance!
FYI - here is what worked. Evidently I had a syntax error...
inputs.conf
[tcp://12345]
connection_host = dns
index = foo
sourcetype = foo_log
props.conf
TRANSFORMS-1_extract_hostname = foo_host
TRANSFORMS-2_overide_foo_index = foo_index_override
TRANSFORMS-3_overide_foo_sourcetype = foo_sourcetype_override
transforms.conf
[foo_host]
DEST_KEY = MetaData:Host
FORMAT = host::$1
[foo_index_override]
SOURCE_KEY = MetaData:Host
REGEX = host::abc123.+
DEST_KEY = _MetaData:Index
FORMAT = bar
[foo_sourcetype_override]
SOURCE_KEY = MetaData:Host
REGEX = host::abc123.+
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::bar_logs
If the distinguishing characteristic is the host field then REGEX will not find it as it looks at _raw. Try putting this in props.conf on the HF:
[host::abcd-1234-efgh-blahblah.com]
Transforms-OverRide-Foo-index = OverRide-Foo
Transforms-OverRide-Foo-sourcetype = OverRide-Foo_log
[host::zyxw-9876-ghyh-blahblah.com]
Transforms-OverRide-Foo-index = OverRide-Foo
Transforms-OverRide-Foo-sourcetype = OverRide-Foo_log
Then transforms.conf becomes
[OverRide-Foo]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = bar
[OverRide-Foo_Log]
REGEX = .
DEST_KEY = MetaData:Sourcetype
FORMAT = bar_log
FYI - here is what worked. Evidently I had a syntax error...
inputs.conf
[tcp://12345]
connection_host = dns
index = foo
sourcetype = foo_log
props.conf
TRANSFORMS-1_extract_hostname = foo_host
TRANSFORMS-2_overide_foo_index = foo_index_override
TRANSFORMS-3_overide_foo_sourcetype = foo_sourcetype_override
transforms.conf
[foo_host]
DEST_KEY = MetaData:Host
FORMAT = host::$1
[foo_index_override]
SOURCE_KEY = MetaData:Host
REGEX = host::abc123.+
DEST_KEY = _MetaData:Index
FORMAT = bar
[foo_sourcetype_override]
SOURCE_KEY = MetaData:Host
REGEX = host::abc123.+
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::bar_logs
is there another way to write the props / transforms to look at the tcp source and regex _raw , for the hostname or IP , like
Props.conf
[source::tcp:12345]
Transforms-OverRide-Foo-index = OverRide-Foo
Transforms-OverRide-Foo-sourcetype = OverRide-Foo_log
transforms.conf
[OverRide-Foo]
REGEX = (abcd* | 1.2.3.4 )
DEST_KEY = _MetaData:Index
FORMAT = bar
[OverRide-Foo_Log]
REGEX = (abcd* | 1.2.3.4 )
DEST_KEY = MetaData:Sourcetype
FORMAT = bar_log
Try adding this to your transforms.conf stanzas
SOURCE_KEY = metadata:host
still no luck with
SOURCE_KEY = MetaData:Host
Thank you.
I followed your instructions, but it did not work...
I don't have any conflicts in .../system/local, and I made the change in .../etc/apps/search/local < where the input was created. Should I move it to ...system/local?
I believe your config is correct, but something must be conflicting or the HF does not see the Host =abcd-1234-efgh-blahblah.com
in the _raw log.
I tried changing the host to the IP and still no luck.
Any ideas ?
I will keep looking for conflicts as there is so much old tangled garbage everywhere in this deployment.
Thank you
There's no need to move the configuration to etc/system/local.
You installed the config files on the HF and then restarted the HF, right?
The value following host:: should be whatever appears in the host field when you search for events from that server.
btool can help you figure out where configurations are being set.
splunk btool --debug inputs list | more
yes, I always restart after a wq of the conf.
This is weird I don't see anything abnormal with btool.
I am wondering if there is something funky with the logs the HF receives, like its not seeing the host field but I know the the name and/or ip is in the raw event.