Filter events from syslog

[gcusello]

Hi at all,
this is a recursive question which I often I answered in past!

I have to filter before indexing logs received by syslog: I have to take some events and discard the others:

• I have a Load Balancer and two Heavy Forwarders that receive send logs and forward them to two Indexers.
• On both indexers I inserted in props.conf [rsa_sa] TRANSFORMS-set-rsa_sa=set_discard,set_parse

• On both indexers I inserted in transforms.conf
  [set_parse]
  REGEX = |AUTHENTICATION|(Logon|Logoff)
  DEST_KEY = queue
  FORMAT = indexQueue
  [set_discard]
  REGEX = .
  DEST_KEY = queue
  FORMAT = nullQueue

• I restarted Indexers

• I continue to have all the events!

Regex is correct: I tested it in Splunk search and regex101.com, anyway these are two events: the first to take and the second to discard;

Jan 19 11:20:57 xxx.xx.xx.xxx Jan 19 2018 10:21:31 rsasa CEF:0|RSA|Security Analytics Audit|10.6.5.0|AUTHENTICATION|Logon|6|rt=Jan 19 2018 10:21:31 suser=xxxxxx sourceServiceName=SA_SERVER deviceExternalId=xxxxxxxxxxxxxxxxxx deviceProcessName=SA_SERVER outcome=Success
Jan 19 12:20:16 xxx.xx.xx.xxx Jan 19 2018 11:20:50 rsahybridlog CEF:0|RSA|Security Analytics Audit|10.6.5.0|DATA_ACCESS|sdk.values|6|rt=Jan 19 2018 11:20:50 src=xxx.xx.xx.xxx spt=55350 suser=xxxxx sourceServiceName=CONCENTRATOR deviceExternalId=xxxxxxxxxxxxxxxxxxxxxxxxxxx deviceProcessName=NwConcentrator outcome=pending msg=has issued values (channel 422927) (thread 35217)

I'm using Splunk 7.0.0.

Where could I search the problem?

Thank you in advance.

Bye.
Giuseppe

[harsmarvania57 · Accepted solution]

Hi @cusello,

If you are receiving logs on Heavy Forwarders first and then Heavy Forwarders sending it to Indexers, in this case those props.conf and transforms.conf should be on Heavy Forwarders not on Indexers because parsing already completed on Heavy Forwarder so your configuration on Indexers will not do any parsing again.

[FrankVl]

Shouldn't this config go on the Heavy Forwarders? And even if that wouldn't be necessary, it would still be beneficial to put it there, right, as that drops the events before being sent across to the indexers.

[mayurr98]

can you just change TRANSFORMS-set-rsa_sa to TRANSFORMS-set I do not think this will do any changes but just check! Also one more question, which add-on you are using to get these logs? Cause if you are using any add-on then do check for sourcetype rename's as it happened in palo_alto_logs where palo_log changed to palo:log see default/props.conf for more.