topic Sending data to nullqueue using props and transafoms is not working. in Splunk Enterprise

Sending data to nullqueue using props and transafoms is not working.

SagarSplunk — Tue, 29 Sep 2020 15:01:36 GMT

Hi All,

I am trying to send data to nullqueue so that events will not get indexed. we can save license consumption.

Props.conf

[testfiltering]
DATETIME_CONFIG =
NO_BINARY_CHECK = true
category = Custom
disabled = false
pulldown_type = true
TRANSFORMS-SERVICE = eventsDrop

transforms.conf

[eventsDrop]
REGEX = (?m)^THREAD.SERVICE-.*E2ELoggingSupport.
DEST_KEY = queue
FORMAT = nullQueue

Log details to be filtered
2017-07-05 15:54:30.157 INFO THREAD-1321 SERVICE-[MDP Feeder]_BusinessFlowSelectorService_H075F54304221O1P H075F54304321O1Q E2ELoggingSupport : Payment Id: H075F54304321O1Q, JMS msg received header [Destination=queue:///GPP.FROMDP.SEND.PAYMNT.INSTRCTN.IN,DeliveryMode=2,Expiration=0 null,Priority=4,MessageID=ID:414d51204445564750503032202020205959ceef1000b103,Timestamp=1499233913142 2017-07-05T15:51:53.142,CorrelationID=null,ReplyTo=null,Redelivered=false,Type=null] PropertyNames=[JMS_IBM_Format=MQSTR ][JMS_IBM_Character_Set=UTF-8][JMSXDeliveryCount=1][JMS_IBM_Encoding=273][JMSXUserID=pegapsup ][JMS_IBM_MsgType=8][JMS_IBM_PutApplType=28][JMS_IBM_PutDate=20170705][JMS_IBM_PutTime=05512391][JMSXAppID=hermes.browser.HermesBrowser]

Thanks/Sagar

Re: Sending data to nullqueue using props and transafoms is not working.

s2_splunk — Thu, 20 Jul 2017 21:31:09 GMT

Your RegEx anchors "THREAD" to the beginning of the line, but it doesn't show up at the beginning of the line. Either add the patterns for timestamp and category to your RegEx or remove the caret (^).

Also, make sure you put those configs where the parsing occurs; probably your indexers.

Re: Sending data to nullqueue using props and transafoms is not working.

SagarSplunk — Fri, 21 Jul 2017 03:57:12 GMT

HI SSievert,
Now I changed my configurations as below but still I am unable to filter out the above events. am I missing something? syntax is correct for regex? I trying to filter out events before it index

[eventsDrop]
REGEX = SERVICE-.E2ELoggingSupport.
DEST_KEY = queue
FORMAT = nullQueue

Thanks/Sagar

Re: Sending data to nullqueue using props and transafoms is not working.

s2_splunk — Tue, 29 Sep 2020 15:00:17 GMT

You need an "*" after the first "." to match on more than just one character. You can also skip the last "." Try this:
[eventsDrop]
REGEX = SERVICE-.*E2ELoggingSupport
DEST_KEY = queue
FORMAT = nullQueue
Drop it on your indexer and restart Splunk.

BTW, RegExr is a good tool to test whether your RegEx constructs work. 😉

Re: Sending data to nullqueue using props and transafoms is not working.

SagarSplunk — Sat, 22 Jul 2017 06:18:20 GMT

Hi SSievert

I tried above Regex its too not working for me are there limitation for free version of splunk.

Re: Sending data to nullqueue using props and transafoms is not working.

s2_splunk — Sat, 22 Jul 2017 18:03:26 GMT

There are limitations in the free version of Splunk, but this is not one of them.
If you make sure that

the stanza name in props.conf matches your sourcetype and
the stanza name in transforms matches what you used after TRANSFORMS-xxxx= and
your RegEx works and matches what you want to match and
you deploy props/transforms in the right place (where parsing happens, i.e. indexer or heavy forwarder, NOT universal forwarder)
you restart splunk or reload the configuration after making the change

this will work with any version of Splunk Enterprise.