I want to keep specific events which contains few strings in event but around 30 OR statement I have to write in regex to keep these events like below my transforms.conf-
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = (\|abcType=|\|abcId=|KType=Sys,\sDate=|abc\sxyz.exe\sZzzz|abc|xyz|pqrs|\JType=test,\sStats=ok)
DEST_KEY = queue
FORMAT = indexQueue
below are few strings i am checking in regex-
|abcType=
|abcId=
KType=Sys, Date=
abc xyz.exe Zzzz
abc
xyz
pqrs
JType=test, Stats=ok
These are 30 strings which I am checking in event so I want to know how feasible it will be or is there any better option?
The correct answer is to always filter as far back in the pipeline as possible. If you can change the software that is writing the logs, do that. If you can edit the logs after it writes before Splunk grabs them, do that. If you have a HF layer, do it there. If you only have Indexers, then do it there. If you cannot do it there, then do it in SPL. So if you CAN do it farther back then do so because eventually your Indexers are going to be overwhelmed and you are going to have to buy more and you should put that off as long as possible.
P.S. Your \JType
should be just JType
.
Hi ips_mandar,
what's your problem?
in your regex I see just a few modifications (= is a special char to escape).
REGEX = \|abcType\=|\|abcId\=|KType\=Sys,\sDate\=|abc\sxyz.exe\sZzzz|abc|xyz|pqrs|\JType\=test,\sStats\=ok
remember that in props.conf you have to put before the command with REGEX = .
and than the other
[your_sourcetype]
TRANSFORMS-null= setnull,setparsing
for additional information see at https://docs.splunk.com/Documentation/Splunk/7.3.0/Forwarding/Routeandfilterdatad
Bye.
Giuseppe
Thanks @gcusello
My problem is I want to know how efficient this method than to write python scripts to filter the events...currently it is working with mentioned stanzas on sample log files.
and I could not understand statement-" in props.conf you have to put before the command with REGEX = . and than the other"
My main goal is to know will it be efficient way to do this for zip files on windows since I am writing around 30 regex combination to index required events.
Hi ips_mandar,
I usually use this way to dinamically set the hostname.
There's another choice modifying hostname before indexing on Indexers, but I prefer on Forwarders.
about props.conf, you need of a command on props.conf to enable the transforms.conf stanza as you can read in the link I sent.
In other words:
props.conf
[your_sourcetype]
TRANSFORMS-set= setnull,setparsing
transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = \|abcType\=|\|abcId\=|KType\=Sys,\sDate\=|abc\sxyz.exe\sZzzz|abc|xyz|pqrs|\JType\=test,\sStats\=ok
DEST_KEY = queue
FORMAT = indexQueue
Bye.
Giuseppe
Thanks Cusello
My question is more on efficiency and yes I am making all these props.conf and transforms.conf changes on HF.
reason behind these stanzas is to reduce license usage by removing unwanted data.
Hi ips_mandar,
Ok, you have to insert them on the HFs and on Indexers (if you have logs that directly arrive to them).
If you're satisfied by this answer, please accept and/or upvote it.
Bye, see the next time.
Giuseppe