topic Re: Help troubleshooting why events are not filtered to nullQueue with my props.conf and transforms.conf ? in Getting Data In

Help troubleshooting why events are not filtered to nullQueue with my props.conf and transforms.conf ?

tomcochran — Wed, 29 Jun 2016 16:26:45 GMT

The input is working and the events are getting to Splunk. I am trying to get a filter going to drop noisy events. I have created an app that is deployed via a configuration server. I have tried this many different ways, but it doesn't seem to drop the events. The app is being pushed to the Universal Forwarder on Windows. The regex matches on online tools.

props.conf

[SAP_portal_security_audit]
TRANSFORMS-set = discard_events

transforms.conf

 [discard_events]
 REGEX = (ACCESS\.ERROR|USERMAPPING\.USE)
 DEST_KEY = queue
 FORMAT = nullQueue

Log data

.0#2016 06 29 08:54:55:906#0-500#Info#/System/Security/Audit/UserMapping#
#BC-JAS-SEC-UME#com.sap.security.core.sda#C0000A980ACE0ACF0000000600003AE4#28973850000000004#sap.com/irj#com.sap.security.core.util.SecurityAudit#Guest#0##90B5B81E3DA611E6B986000001BA1B1A#90b5b81e3da611e6b986000001ba1b1a#90b5b81e3da611e6b986000001ba1b1a#0#Thread[pool-2141-thread-1,5,Dedicated_Application_Thread]#Plain##
User mapping used   | USERMAPPING.USE   | USER.PRIVATE_DATASOURCE.un:eServices  |   | systemtype=[SAP_CRM], system=["SAP_CRM" (system landscape: "EnterprisePortal")], remote user ID=[ZESRVUSER], uses strong encryption=[true]#


#2.0#2016 06 29 10:41:50:911#0-500#Warning#/System/Security/Audit/Access#
#EP-KM-FWK-RF#sap.com/com.sap.netweaver.bc.rf#C0000A989048C7010000000400000EA0#2778350000000004#sap.com/eServicesMasthead#com.sapportals.wcm.repository.security.SecurityAudit$AccessLog#UKSHEPA#16689##FCBF8AB33E0F11E6CA770000002A64EE#fb491e783e0f11e6c3fb0000002a64ee#fb491e783e0f11e6c3fb0000002a64ee#0#Thread[1047252450|pcd:portal_content/bungeContent/protectedContent/authenticatedDesktop/frameworkPages/authenticatedFrameworkPage/eServicesLightMasthead\#com%2esap%2eportal%2enavigation%2eportallauncher%2edefault.pcd%3aportal_content%2fbungeContent%2fprotectedContent%2fauthenticatedDesktop%2fframeworkPages%2fauthenticatedFrameworkPage.eServicesLightMasthead,5,Managed_Application_Thread]#Plain##
UKSHEPA | ACCESS.ERROR  | /documents/Public Documents/AgSite/SiteImages/b4b99ce4da58004ef7e8614edb99e3d2.xml    | leaf_write_content,leaf_write_properties#
    6/29/16 
11:03:07.943 AM 
#2.0#2016 06 29 11:03:07:943#0-500#Warning#/System/Security/Audit/Access#
#EP-KM-FWK-RF#sap.com/com.sap.netweaver.bc.rf#C0000A9890453995000000CE00003A30#6820051000000004#sap.com/eServicesPublic#com.sapportals.wcm.repository.security.SecurityAudit$AccessLog#Guest#0##F59B7BD13E1211E6AE430000006810D3#f59b7bd13e1211e6ae430000006810d3#f59b7bd13e1211e6ae430000006810d3#0#Thread[1784794969|pcd:portal_content/bungeContent/publicContent/roles/eServices_Home/publicHome/locationInfoTabbedContainer/PersonnelSummary\#com%2esap%2eportal%2enavigation%2eportallauncher%2eanonymous.pcd%3aportal_content%2fbungeContent%2fpublicContent%2fanonymousLightDesktop%2fframeworkPages%2fanonymousLightFramework.com%2esap%2eportal%2elightinnerpage.com%2esap%2eportal%2elightcontentarea.content.locationInfoTabbedContainer.PersonnelSummary,5,Managed_Application_Thread]#Plain##
Guest   | ACCESS.ERROR  | /documents/Public Documents/AgSite/Personnel/9f40e230b555f773b47ffb300514e66e.xml | leaf_write_content,leaf_write_properties#

Re: Help troubleshooting why events are not filtered to nullQueue with my props.conf and transforms.conf ?

hgrow — Thu, 30 Jun 2016 05:26:16 GMT

Hi tomcochran,

i guess you have deployed the configuration above on your forwarder? If so, thats the problem (i assume your configuration is correct) !

The UniversalForwarder does not parse any of the events he is forwarding. Just bring the configuration to your indexer and it should work.

I hope it helps.

Greetings

Re: Help troubleshooting why events are not filtered to nullQueue with my props.conf and transforms.conf ?

gcusello — Wed, 30 Sep 2020 02:53:46 GMT

I solved a similar issue configuring two stanzas in transforms.conf and inserting two commands in props.conf.
note that it's important the order between the two commands in props.conf (the order of stanzas in transforms.conf isn't important):

to take only a set of logs discarding the others: before nullqueue and after the log stanza
to take all the logs discarding only a set of logs before the log stanza and after null queue

in other words: before the command with all the logs (REGEX=.) and after the command with the set of logs you want (REGEX=xxxx).

see the following example where set_AS are the logs I want to index, nullqueue are the logs I want to discard.

in props.conf
TRANSFORMS-set-AS=set_nullqueue,set_AS

in transforms.conf

nullqueue

[set_nullqueue]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

AS

[set_AS]
REGEX=SRVE
DEST_KEY = queue
FORMAT = indexQueue

Bye.

Giuseppe

Re: Help troubleshooting why events are not filtered to nullQueue with my props.conf and transforms.conf ?

tomcochran — Thu, 30 Jun 2016 13:59:59 GMT

Thank you, this worked. So here is a question regarding License, Do the events getting dropped at the indexer count towards the capacity? Is it capacity getting to the indexer, or capacity of data indexed?

Re: Help troubleshooting why events are not filtered to nullQueue with my props.conf and transforms.conf ?

hgrow — Thu, 30 Jun 2016 14:09:01 GMT

Hi tomcochran,

i'm glad i could help. I made my comment an answer. Can you do me a favor and accept it?

To answer your license questions, it's capacity of data indexed. The events send to nullqueue wont stress your licence.

Greetings
hgrow

Re: Help troubleshooting why events are not filtered to nullQueue with my props.conf and transforms.conf ?

tomcochran — Thu, 30 Jun 2016 18:32:09 GMT

Awesome, thank you so much.

Re: Help troubleshooting why events are not filtered to nullQueue with my props.conf and transforms.conf ?

henrikstorm — Tue, 29 Sep 2020 23:58:29 GMT

I have tried

props.conf
[hfss_source]
TRANSFORMS-set= set_parsing,set_null

transforms.conf
[set_null]
REGEX = standby
DEST_KEY = queue
FORMAT = nullQueue

[set_parsing]
REGEX =.
DEST_KEY=queue
FORMAT=indexQueue

Trying to get rid of all entries with the word "standby" in it, but everything gets indexed.

splunk cmd btool props list hfss_source
shows correctly, but

splunk cmd btool transforms list hfss_source
shows absolutely nothing

Any ideas ? I am completely lost, I feel I have done what everybody is writing will work, but I just can't get it to work?

Any help would be greatly appriciated