How to filter out the logs during indexing using regex?

[anandhalagaras1]

Hi Team,

I want to filter out the logs during the indexing level itself i.e. If the event comes with the following format as mentioned below with GET / - 111 then the logs should not be ingested into Splunk.

So kindly help out with the props and transforms for the same.

Sample Event:

2020-07-22 12:53:53 xx.xxx.xx.xx GET / - 111 - xx.xxx.x.xxx - - xxx x x xx

[to4kawa]

https://docs.splunk.com/Documentation/Splunk/8.0.5/Forwarding/Routeandfilterdatad

transforms.conf

[null]
REGEX = GET\s/\s-\s111
DEST_KEY = queue
FORMAT = nullQueue

[anandhalagaras1]

@to4kawa ,

Eventhough i used your props and transforms i am can still see the logs are getting ingested into Splunk. Also the way i have setup is that the UF is installed in the client machines and from there i have mentioned the outputs as the HF and from HF it will reach the Splunk Cloud indexers. So i have placed the props and transforms in the HF as you have mentioned but still I can see the logs are getting parsed into Search head.

Sample format of logs for reference. So if the below "GET / - 111" is seen then the logs should not be ingested into Splunk. So kindly help with the props and transforms.

2020-07-24 02:27:38 09.876.543.21 GET / - 111 - 12.345.6.789 - - 123 4 5 67
2020-07-24 02:27:34 09.876.543.21 GET / - 111 - 12.345.6.789 - - 123 4 5 67
2020-07-24 02:27:29 09.876.543.21 GET / - 111 - 12.345.6.789 - - 123 4 5 6
2020-07-24 02:27:29 09.876.543.21 GET / - 111 - 12.345.6.789 - - 123 4 5 6
2020-07-24 02:27:23 09.876.543.21 GET / - 111 - 12.345.6.789 - - 123 4 5 67
2020-07-24 02:27:19 09.876.543.21 GET / - 111 - 12.345.6.789 - - 123 4 5 67

[anandhalagaras1]

Eventhough i used your props and transforms i am can still see the logs are getting ingested into Splunk. Also the way i have setup is that the UF is installed in the client machines and from there i have mentioned the outputs as the HF and from HF it will reach the Splunk Cloud indexers. So i have placed the props and transforms in the HF as you have mentioned but still I can see the logs are getting parsed into Search head.

Sample format of logs for reference. So if the below "GET / - 111" is seen then the logs should not be ingested into Splunk. So kindly help with the props and transforms.

2020-07-24 02:27:38 09.876.543.21 GET / - 111 - 12.345.6.789 - - 123 4 5 67
2020-07-24 02:27:34 09.876.543.21 GET / - 111 - 12.345.6.789 - - 123 4 5 67
2020-07-24 02:27:29 09.876.543.21 GET / - 111 - 12.345.6.789 - - 123 4 5 6
2020-07-24 02:27:29 09.876.543.21 GET / - 111 - 12.345.6.789 - - 123 4 5 6
2020-07-24 02:27:23 09.876.543.21 GET / - 111 - 12.345.6.789 - - 123 4 5 67
2020-07-24 02:27:19 09.876.543.21 GET / - 111 - 12.345.6.789 - - 123 4 5 67

So anyone kindly help on my request please to filter out the logs.

[gcusello · Accepted solution]

Hi @anandhalagaras1,

try to add to the regex hinted by @to4kawa a backslash before the slash, in other words try this regex:

GET\s\/\s-\s111

as you can see in https://regex101.com/r/kloFhs/1 if you don't escape slash, the regex doesn't run.

Ciao.

Giuseppe

[anandhalagaras1]

Thank you. It worked like a charm..

[gcusello]

Hi @anandhalagaras1,

good, have a good splunkg!

Ciao and next time.

Giuseppe

P.S.: Karma Points are appreciated 😉