topic Re: Unable to filter specific events with props.conf & transforms.conf in Getting Data In

Unable to filter specific events with props.conf & transforms.conf

marshallsuk — Wed, 30 Sep 2020 04:57:18 GMT

I've been reading into how to filter incoming events between the Parsing and Indexing stages of our data pipeline and found a Splunk doc (https://docs.splunk.com/Documentation/SplunkCloud/8.0.2001/Forwarding/Routeandfilterdatad) which suggests it should be possible to do this by just adding a props.conf and transforms.conf at the indexer:

I have been through and added in what I thought would be valid entries in the conf files, as follows:

props.conf

[source::IIS_Exchange]
TRANSFORMS-set= setnull,ExchangeParsing

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[ExchangeParsing]
REGEX = /(-\s401\s1\s1[39][02][69])/
DEST_KEY = queue
FORMAT = indexQueue

The regex should be monitoring for HTTP 401 errors with a sub-status of 1 and a pair of event error codes we're monitoring for (1326 & 1909), for example:

401 1 1909 9
401 1 1326 67

I've tested the regex in both Notepad++ and on RegExr.com and it looks like it should work.

We use a Universal Forwarder to send the data into the indexer, which is just trying to monitor standard IIS logs on a load balanced server pair; the inputs.conf for the relevant source is as follows:

[monitor://G:\inetpub\logs\LogFiles\W3SVC1*.log]
index = mar
source_type = iis
disabled = false
recursive = true
source = IIS_Exchange

I've restarted Splunk on the indexer for this to take effect but nothing appears to be happening. I had expected that if the 2nd stanza in the transforms.conf was incorrect we'd get no data at all from the source, as it would all initially be filtered to the nullQueue. This suggests to me that the transform isn't applying to the incoming data at all but I cannot fathom why.

Can anybody please tell me what I'm doing wrong?

Re: Unable to filter specific events with props.conf & transforms.conf

richgalloway — Wed, 15 Apr 2020 12:48:22 GMT

Please share some sample events the REGEX is to parse.
The REGEX value appears to have extra / characters on each end. It also will match more than two expected error codes. Did you test it on regex101.com? Make sure your regex tester uses PCRE mode.

source_type should be sourcetype in inputs.conf.

Re: Unable to filter specific events with props.conf & transforms.conf

marshallsuk — Wed, 15 Apr 2020 13:52:27 GMT

Hi,

Here's a few sanitised examples of the data we're looking at (these all matched on RegExr.com & I've double checked them on the site you linked and they also seem to match):

2020-04-14 11:57:00 123.123.123.123 POST /Microsoft-Server-ActiveSync/default.eas Cmd=Sync&User=UsernameHere&DeviceId=DeviceIDHere&DeviceType=SamsungDevice&CorrelationID=;&cafeReqId=UniqueIDHere; 443 UsernameHere 123.123.123.123 - 401 1 1909 9
2020-04-14 11:56:47 123.123.123.123 POST /Microsoft-Server-ActiveSync/default.eas Cmd=Sync&User=UsernameHere&DeviceId=DeviceIDHere&DeviceType=SamsungDevice&CorrelationID=;&cafeReqId=UniqueIDHere; 443 UsernameHere 123.123.123.123 - 401 1 1909 51
2020-04-14 11:55:37 123.123.123.123 POST /Microsoft-Server-ActiveSync/default.eas Cmd=FolderSync&User=UsernameHere&DeviceId=DeviceIDHere&DeviceType=Android&CorrelationID=;&cafeReqId=UniqueIDHere; 443 UsernameHere 123.123.123.123 - 401 1 1326 67
2020-04-14 11:54:17 123.123.123.123 POST /Microsoft-Server-ActiveSync/default.eas Cmd=FolderSync&User=UsernameHere&DeviceId=DeviceIDHere&DeviceType=Android&CorrelationID=;&cafeReqId=UniqueIDHere; 443 UsernameHere 123.123.123.123 - 401 1 1326 33
2020-04-14 11:54:15 123.123.123.123 POST /Microsoft-Server-ActiveSync/default.eas Cmd=FolderSync&User=UsernameHere&DeviceId=DeviceIDHere&DeviceType=Android&CorrelationID=;&cafeReqId=UniqueIDHere; 443 UsernameHere 123.123.123.123 - 401 1 1326 49
2020-04-14 11:53:12 123.123.123.123 POST /Microsoft-Server-ActiveSync/default.eas Cmd=FolderSync&User=UsernameHere&DeviceId=DeviceIDHere&DeviceType=Android&CorrelationID=;&cafeReqId=UniqueIDHere; 443 UsernameHere 123.123.123.123 - 401 1 1326 33
2020-04-14 11:53:11 123.123.123.123 POST /Microsoft-Server-ActiveSync/default.eas Cmd=FolderSync&User=UsernameHere&DeviceId=DeviceIDHere&DeviceType=Android&CorrelationID=;&cafeReqId=UniqueIDHere; 443 UsernameHere 123.123.123.123 - 401 1 1326 51

The extra / characters signify the start/end of a regex; I'll remove them if they're not required.

Thanks for the correction on the inputs.conf setting; I'll go in and amend that ASAP!

Re: Unable to filter specific events with props.conf & transforms.conf

richgalloway — Wed, 15 Apr 2020 16:55:29 GMT

Delimiters should not be used in REGEX unless part of the text being matched.
This regex works with your sample data. Perhaps you'll get different results with it. -\s401\s1\s(?:1326|1909).