Getting Data In

How to blacklist using props.conf and transforms.conf?

splunkcol
Builder

I need to reject or not index the logs that have the word "notice" inside the log

I understand that it is done using these two files

I have 2 doubts:

1. Is the regex ok?
2. If the path is constantly changing I can use a wildcard? [source::/folder/folder/logs/firewall-xxxxx/* ]

props.conf

[source::/folder/folder/logs/firewall-xxxxx/2020/12/4/local7.log]
TRANSFORMS-null= setnull

transforms.conf

[setnull]
REGEX = notice
DEST_KEY = queue
FORMAT = nullQueue

Sample Log

date=2019-05-10 time=11:37:47 logid="0000000013" type="traffic" subtype="forward" level="notice" vd="vdom1" eventtime=1557513467369913239 srcip=10.1.100.11 srcport=58012 srcintf="port12" srcintfrole="undefined" dstip=23.59.154.35 dstport=80 dstintf="port11" dstintfrole="undefined" srcuuid="ae28f494-5735-51e9-f247-d1d2ce663f4b" dstuuid="ae28f494-5735-51e9-f247-d1d2ce663f4b" poluuid="ccb269e0-5735-51e9-a218-a397dd08b7eb" sessionid=105048 proto=6 action="close" policyid=1 policytype="policy" service="HTTP" dstcountry="Canada" srccountry="Reserved" trandisp="snat" transip=172.16.200.2 transport=58012 appid=34050 app="HTTP.BROWSER_Firefox" appcat="Web.Client" apprisk="elevated" applist="g-default" duration=116 sentbyte=1188 rcvdbyte=1224 sentpkt=17 rcvdpkt=16 utmaction="allow" countapp=1 osname="Ubuntu" mastersrcmac="a2:e9:00:ec:40:01" srcmac="a2:e9:00:ec:40:01" srcserver=0 utmref=65500-742

 
 

splunkcol_0-1607090305117.png

 

0 Karma
1 Solution

twinspop
Influencer

The regex is simple. You'd have to answer if it's appropriate or not. Any log event with "notice" anywhere in the event will match.  

Pattern matching in the `[source::]` qualifier works like it does with inputs. `*` matches anything but file delimiters, and `...` matches anything. Something like this might work   

`[source::/folder/folder/logs/firewall-*/*/*/*/local*.log]`

View solution in original post

splunkcol
Builder

If someone is helpful, this only applies to Heavy forwarders

In Universal forwarder there is no filtering capability through regular expressions

0 Karma

twinspop
Influencer

The regex is simple. You'd have to answer if it's appropriate or not. Any log event with "notice" anywhere in the event will match.  

Pattern matching in the `[source::]` qualifier works like it does with inputs. `*` matches anything but file delimiters, and `...` matches anything. Something like this might work   

`[source::/folder/folder/logs/firewall-*/*/*/*/local*.log]`

verbal_666
Builder

Hi.

This is a simple "positive/included" regex. OK!

Is there a way, in reverse, to do a "negative/exclude" way to filter data?

As an example, as for the log of the user,
I DO NOT WANT "notice" PATTERN to be indexed, but all the rest,
something like

REGEX != notice

 

Is there a simple way inside props/transforms?

0 Karma

verbal_666
Builder

I resolved this issue with 2 (or more) transformations, dropping all unuseful events...

props.conf

[mysourcetype]
TRANSFORMS-filter = drop

transforms.conf

[drop]
REGEX = drop_event1|drop_event2|drop_eventX
DEST_KEY = queue
FORMAT = nullQueue

 

I think it's the best way. Maybe the only one 🤔

 

But, at the same time, there's is no way to make both work with drop and get transformation,

props.conf

[mysourcetype]
TRANSFORMS-filter = drop,filter

transforms.conf

[drop]
REGEX = drop_event1|drop_event2|drop_eventX
DEST_KEY = queue
FORMAT = nullQueue

[filter]
REGEX = get_event1|get_event2|get_eventX
DEST_KEY = queue
FORMAT = indexQueue

I would like to explain Splunk 8,

  1. FIRST: drop all events containing pattern regex "drop_event1|drop_event2|drop_eventX"
  2. SECOND: get only events containing pattern regex "get_event1|get_event2|get_eventX"

Splunk, after dropping, get all (".*") except "drop_event1|drop_event2|drop_eventX" 😪

Any suggestion?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...