Splunk Search

Feasibility of regex to keep specific events and discard the rest

ips_mandar
Builder

I want to keep specific events which contains few strings in event but around 30 OR statement I have to write in regex to keep these events like below my transforms.conf-

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = (\|abcType=|\|abcId=|KType=Sys,\sDate=|abc\sxyz.exe\sZzzz|abc|xyz|pqrs|\JType=test,\sStats=ok)
DEST_KEY = queue
FORMAT = indexQueue

below are few strings i am checking in regex-

|abcType=
|abcId=
KType=Sys, Date=
abc xyz.exe Zzzz
abc
xyz
pqrs
JType=test, Stats=ok

These are 30 strings which I am checking in event so I want to know how feasible it will be or is there any better option?

0 Karma

woodcock
Esteemed Legend

The correct answer is to always filter as far back in the pipeline as possible. If you can change the software that is writing the logs, do that. If you can edit the logs after it writes before Splunk grabs them, do that. If you have a HF layer, do it there. If you only have Indexers, then do it there. If you cannot do it there, then do it in SPL. So if you CAN do it farther back then do so because eventually your Indexers are going to be overwhelmed and you are going to have to buy more and you should put that off as long as possible.

P.S. Your \JType should be just JType.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi ips_mandar,
what's your problem?
in your regex I see just a few modifications (= is a special char to escape).

REGEX = \|abcType\=|\|abcId\=|KType\=Sys,\sDate\=|abc\sxyz.exe\sZzzz|abc|xyz|pqrs|\JType\=test,\sStats\=ok

remember that in props.conf you have to put before the command with REGEX = . and than the other

[your_sourcetype]
TRANSFORMS-null= setnull,setparsing

for additional information see at https://docs.splunk.com/Documentation/Splunk/7.3.0/Forwarding/Routeandfilterdatad

Bye.
Giuseppe

ips_mandar
Builder

Thanks @gcusello
My problem is I want to know how efficient this method than to write python scripts to filter the events...currently it is working with mentioned stanzas on sample log files.
and I could not understand statement-" in props.conf you have to put before the command with REGEX = . and than the other"
My main goal is to know will it be efficient way to do this for zip files on windows since I am writing around 30 regex combination to index required events.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi ips_mandar,
I usually use this way to dinamically set the hostname.
There's another choice modifying hostname before indexing on Indexers, but I prefer on Forwarders.
about props.conf, you need of a command on props.conf to enable the transforms.conf stanza as you can read in the link I sent.
In other words:

props.conf

[your_sourcetype]
TRANSFORMS-set= setnull,setparsing

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \|abcType\=|\|abcId\=|KType\=Sys,\sDate\=|abc\sxyz.exe\sZzzz|abc|xyz|pqrs|\JType\=test,\sStats\=ok
DEST_KEY = queue
FORMAT = indexQueue

Bye.
Giuseppe

0 Karma

ips_mandar
Builder

Thanks Cusello
My question is more on efficiency and yes I am making all these props.conf and transforms.conf changes on HF.
reason behind these stanzas is to reduce license usage by removing unwanted data.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi ips_mandar,
Ok, you have to insert them on the HFs and on Indexers (if you have logs that directly arrive to them).
If you're satisfied by this answer, please accept and/or upvote it.

Bye, see the next time.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...