How to use Regular Expression in props.conf for source matching?

[cameronjust]

Hi All,

I've got a generic syslog app which pulls in EVERYTHING in the syslog directory with the sourcetype=syslog-unconfigured

inputs.conf

[monitor:///var/log/syslog-ng/*/messages]
index = syslog
sourcetype = syslog:unconfigured
host_segment = 4

This is done so we can catch any new syslog devices that were not configured to go to the correct sourcetype. We have a props.conf that routes data to the right index/sourcetype depending on the hostname.

props.conf

# InfoBlox
[source::/var/log/syslog-ng/(10.164.55.55|10.9.55.56|prodinfoblox1|prodinfoblox2)/messages]
TRANSFORMS-reroute_index = route_to_index_infoblox
TRANSFORMS-reroute_sourcetype = route_to_sourcetype_infoblox:file
TZ=UTC

transforms.conf

[route_to_index_infoblox]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = infoblox

[route_to_sourcetype_infoblox:file]
REGEX = .
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::infoblox:file

Now the above props.conf with a regex for matching on the host in the source doesn't work. However naming each individually does or with a basic wildcard

# InfoBlox
[source::/var/log/syslog-ng/10.164.55.55/messages]
TRANSFORMS-reroute_index = route_to_index_infoblox
TRANSFORMS-reroute_sourcetype = route_to_sourcetype_infoblox:file
TZ=UTC

[source::/var/log/syslog-ng/10.9.55.56/messages]
TRANSFORMS-reroute_index = route_to_index_infoblox
TRANSFORMS-reroute_sourcetype = route_to_sourcetype_infoblox:file
TZ=UTC

[source::/var/log/syslog-ng/prodinfoblox*/messages]
TRANSFORMS-reroute_index = route_to_index_infoblox
TRANSFORMS-reroute_sourcetype = route_to_sourcetype_infoblox:file
TZ=UTC

I've tried escaping the slashes but that doesn't work either.

# This also doesn't work 
[source::\/var\/log\/syslog-ng\/(10.164.55.55|10.9.55.56|prodinfoblox1|prodinfoblox2)\/messages]

Anyone have any ideas how to get the regex to work in the source:: stanza?

Some of these devices have up to 30 hosts and having it all as a one liner would make things much cleaner.

Also I'm aware I can do this in transforms.conf with something like this but then I'd need the source match in two spots which is prone to user error.

[route_to_index_infoblox]
SOURCE_KEY = Metadata:Source
REGEX = \/var\log\syslog\/(192.168.1.1|192.168.1.2|etc.)
DEST_KEY = _MetaData:Index
FORMAT = infoblox

[route_to_sourcetype_infoblox:file]
SOURCE_KEY = Metadata:Source
REGEX = \/var\log\syslog\/(192.168.1.1|192.168.1.2|etc.)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::infoblox:file

There has to be something just slightly off with my regex.

[isoutamo · Accepted solution]

Hi

if I understand this https://docs.splunk.com/Documentation/Splunk/latest/Data/Specifyinputpathswithwildcards#Wildcard_ove... right, you must use also wildcard "*" if you want to use regular expression style matching. Without it it don't recognise this stanza as a regex.

So can you try this

[source::/var/log/syslog-ng/(10.164.55.55|10.9.55.56|prodinfoblox1|prodinfoblox2)*/messages]

r. Ismo 

[cameronjust]

IT WORKS!!!

Thanks for the clarification. I do recall I saw this a few years ago but it obviously never stuck in my mind. I certainly won't forget it now.