topic Re: Data is not getting filtered with my set regex in transforms.conf? in Splunk Search

Data is not getting filtered with my set regex in transforms.conf?

sandipan11 — Thu, 15 Oct 2015 01:48:39 GMT

I have following set up in props.conf and transforms.conf.

props.conf

[source::/opt/apps/splunk/data/test/*]
TRANSFORMS-set= setnull,setparsing

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = (?:Mkt.*\n)(?s)(.*?)(?:[A-Z])
DEST_KEY = queue
FORMAT = indexQueue

I am ingesting a data file containing following data through a universal forawrder, and I made the above change in /splunk/etc/system/local:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                Sample ABC Report   
                Date:2009-12-01     
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
FileName: xxx.txt
-------------------------------------------------------------------
File Processing Start Time:20091201 07:12:14

This file contains the following payments:                                      
Mkt  Bk    Sender Id    Cntry  Curr         Total Value          Total Records
001  0871  1234          124    124             1140843                  7
001  0700  2341          124    124              399559                  2
001  9875  4567          124    124            7373838                  76
001  0700  6789          124    124              761825                  5
ABC.EM.0072.0005
ABC.TM.0072.0006
CDE.EM.0098.0007

File Processing End Time:20091201 07:12:14
-------------------------------------------------------------------

And my objective is to select only following data from above

001  0871  1234          124    124             1140843                  7
001  0700  2341          124    124              399559                  2
001  9875  4567          124    124            13842408                  76
001  0700  6789          124    124              761825                  5

I checked with https://regex101.com/ and my regular expression is extracting the same.

Please advise what could be possible reason that I am getting all data without filtering it. I restarted the indexer after the change.

Re: Data is not getting filtered with my set regex in transforms.conf?

somesoni2 — Thu, 15 Oct 2015 04:32:24 GMT

The props/transforms are placed on Indexers, correct?

Re: Data is not getting filtered with my set regex in transforms.conf?

sandipan11 — Fri, 16 Oct 2015 05:24:04 GMT

yes I kept in indexer

Re: Data is not getting filtered with my set regex in transforms.conf?

lguinn2 — Fri, 16 Oct 2015 05:51:11 GMT

I suspect that the source is not matching the source specified in your props.conf - remember that the source is case sensitive.

[source::/opt/apps/splunk/data/test/*]
TRANSFORMS-set=setnull,setparsing

However, if it was working, you would have another problem. The setnull transform will send all the events to the nullQueue, thereby deleting them. Why? Because the REGEX (.) matches every event - and you have specified that this transform is applied first. Second, the REGEX is applied to each event individually, while http://regex101.com/ is looking at the text as a whole.

I think this will work better, assuming that you get the source correct:

props.conf

[source::/opt/apps/splunk/data/test/*]
SHOULD_LINEMERGE=false
TRANSFORMS-set= setparsing,setnull

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^\s*\d{3}\s+\d{4}
DEST_KEY = queue
FORMAT = indexQueue

Re: Data is not getting filtered with my set regex in transforms.conf?

sandipan11 — Fri, 16 Oct 2015 19:35:27 GMT

Location is Correct. When I was using my configuration no filtering was happening but now with you set up no data is coming I mean data is not ingesting at all..... as data is not coming so it confirms that source is matching rest condition blocking the data.

Please advise.