Splunk Search

REGEX to discard specific event containing " Allow " but not them followad by 2 specific strings ?

fgilain
Engager

Hello,

i need to find the REGEX to allow me to filter what splunk will index.

As it is firewall Logs, it gererates too much volume, so i want to remove events containing the following pattern :

" Allow " (except if there is "ProxyStrip" or "ProxyDeny" somewhere after in the line)

example :

Would match :
Sep 30 16:43:07 host=firewall-01 program=XXX0292D4BCD PID= facility=local1 level=info : firewall-01 http-proxy[3949]: Allow 3-LAN 2-EXTERNAL tcp 192.168.XXX.2 37.XXX.XXX.66 62745 80 msg="ProxyAllow: HTTP Header content type match" proxy_act="HTTP-Client.WebBlocker" rule_name="image/*" content_type="image/gif" (Out_Users_HTTP-rule)

Would not match :

Sep 30 16:43:07 host=firewall-01 program=XXX0292D4BCD PID= facility=local1 level=info : firewall-01 http-proxy[3949]: Allow 3-LAN 2-EXTERNAL tcp 192.168.XXX.2 37.XXX.XXX.66 62745 80 msg="ProxyStrip: HTTP Header content type match" proxy_act="HTTP-Client.WebBlocker" rule_name="image/*" content_type="image/gif" (Out_Users_HTTP-rule)

Sep 30 16:43:07 host=firewall-01 program=XXX0292D4BCD PID= facility=local1 level=info : firewall-01 http-proxy[3949]: Allow 3-LAN 2-EXTERNAL tcp 192.168.XXX.2 37.XXX.XXX.66 62745 80 msg="ProxyDeny: HTTP Header content type match" proxyact="HTTP-Client.WebBlocker" rulename="image/*" contenttype="image/gif" (OutUsers_HTTP-rule)

Any idea ?

Thanks for your help

Florent

Tags (1)
0 Karma

andreas
Explorer

try this as REGEX for the nullQueue:

\sAllow\s(?!.*(?:ProxyStrip|ProxyDeny))

or if you also want to include the ' msg="' part:

\sAllow\s(?!.*\smsg="(?:ProxyStrip|ProxyDeny))
0 Karma

fgilain
Engager

i want to index only "Deny" + "Allow" (for "Allow" : only if followed by ProxyStrip or ProxyDeny only).

0 Karma

jonuwz
Influencer

This is confusing.

You want to filter out if the regex matches.

So what you're saying is you want lines line Allow ProxyStrip and Allow ProxyDeny to be indexed, but ProxyAllow to be discarded ?

fgilain
Engager

In transforms.conf (local one) :

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = Deny
DEST_KEY = queue
FORMAT = indexQueue

[setnoallow]
REGEX = Allow.(?=ProxyDeny|ProxyStrip)|^(?:(?!Allow).)$
DEST_KEY = queue
FORMAT = nullQueue

In props.conf (local one) :

[source::/Logs/Firewall-logs/firewall-01/firewall-01_local0.log]
TRANSFORMS-noallow= setnoallow

[source::/Logs/Firewall-logs/firewall-01/firewall-01_local1.log]
TRANSFORMS-noallow= setnoallow

0 Karma

jonuwz
Influencer
"Allow.*(?=ProxyDeny|ProxyStrip)|^(?:(?!Allow).)*$"

Match "Allow followed by ProxyDeny or ProxyStrip" or "Any line without Allow"

Test :

| stats count 
| eval msg="Allow skldfjlksdjflksdjflksjdf ProxyDeny,Allow skldfjlksdjflksdjflksjdf ProxyAccept,Deny zasdfasdfsdf sdfsdfsdfsdfs"
| fields msg
| makemv delim="," msg
| mvexpand msg
| eval match=if(match(msg,"Allow.*(?=ProxyDeny|ProxyStrip)|^(?:(?!Allow).)*$"),1,0)
0 Karma

jonuwz
Influencer

You probably want to send to the parsingQueue, not the indexQueue too

0 Karma

jonuwz
Influencer

Thats because you're sending the lines that match this to the nullQueue, and sending all the stuff that doesn't match to the real queue.

You need to do

TRANSFORMS-noallow= setnull,setnoallow

and have

[setnoallow]
REGEX = regex_goes here
DEST_KEY = queue
FORMAT = parsingQueue
0 Karma

fgilain
Engager

I still see Allow lines without ProxyDeny or ProxyStrip in the same line when i run a search in the index for last few minutes after restarting splunk.

Maybe did i made a mistake in the way i use transforms.conf ?

0 Karma

fgilain
Engager

ok, i think i got it :

\sAllow\s.+msg="(?!(ProxyStrip|ProxyDeny))

0 Karma

lukejadamec
Super Champion

Your regex seems to work fine in my tester. Are you seeing success in the search string, but not in your index?

0 Karma

fgilain
Engager

Oh no, it still indexes all the "Allow" lines !

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...