Splunk Search

REGEX to discard specific event containing " Allow " but not them followad by 2 specific strings ?

Engager

Hello,

i need to find the REGEX to allow me to filter what splunk will index.

As it is firewall Logs, it gererates too much volume, so i want to remove events containing the following pattern :

" Allow " (except if there is "ProxyStrip" or "ProxyDeny" somewhere after in the line)

example :

Would match :
Sep 30 16:43:07 host=firewall-01 program=XXX0292D4BCD PID= facility=local1 level=info : firewall-01 http-proxy[3949]: Allow 3-LAN 2-EXTERNAL tcp 192.168.XXX.2 37.XXX.XXX.66 62745 80 msg="ProxyAllow: HTTP Header content type match" proxy_act="HTTP-Client.WebBlocker" rule_name="image/*" content_type="image/gif" (Out_Users_HTTP-rule)

Would not match :

Sep 30 16:43:07 host=firewall-01 program=XXX0292D4BCD PID= facility=local1 level=info : firewall-01 http-proxy[3949]: Allow 3-LAN 2-EXTERNAL tcp 192.168.XXX.2 37.XXX.XXX.66 62745 80 msg="ProxyStrip: HTTP Header content type match" proxy_act="HTTP-Client.WebBlocker" rule_name="image/*" content_type="image/gif" (Out_Users_HTTP-rule)

Sep 30 16:43:07 host=firewall-01 program=XXX0292D4BCD PID= facility=local1 level=info : firewall-01 http-proxy[3949]: Allow 3-LAN 2-EXTERNAL tcp 192.168.XXX.2 37.XXX.XXX.66 62745 80 msg="ProxyDeny: HTTP Header content type match" proxyact="HTTP-Client.WebBlocker" rulename="image/*" contenttype="image/gif" (OutUsers_HTTP-rule)

Any idea ?

Thanks for your help

Florent

Tags (1)
0 Karma

Explorer

try this as REGEX for the nullQueue:

\sAllow\s(?!.*(?:ProxyStrip|ProxyDeny))

or if you also want to include the ' msg="' part:

\sAllow\s(?!.*\smsg="(?:ProxyStrip|ProxyDeny))
0 Karma

Engager

i want to index only "Deny" + "Allow" (for "Allow" : only if followed by ProxyStrip or ProxyDeny only).

0 Karma

Influencer

This is confusing.

You want to filter out if the regex matches.

So what you're saying is you want lines line Allow ProxyStrip and Allow ProxyDeny to be indexed, but ProxyAllow to be discarded ?

Engager

In transforms.conf (local one) :

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = Deny
DEST_KEY = queue
FORMAT = indexQueue

[setnoallow]
REGEX = Allow.(?=ProxyDeny|ProxyStrip)|^(?:(?!Allow).)$
DEST_KEY = queue
FORMAT = nullQueue

In props.conf (local one) :

[source::/Logs/Firewall-logs/firewall-01/firewall-01_local0.log]
TRANSFORMS-noallow= setnoallow

[source::/Logs/Firewall-logs/firewall-01/firewall-01_local1.log]
TRANSFORMS-noallow= setnoallow

0 Karma

Influencer
"Allow.*(?=ProxyDeny|ProxyStrip)|^(?:(?!Allow).)*$"

Match "Allow followed by ProxyDeny or ProxyStrip" or "Any line without Allow"

Test :

| stats count 
| eval msg="Allow skldfjlksdjflksdjflksjdf ProxyDeny,Allow skldfjlksdjflksdjflksjdf ProxyAccept,Deny zasdfasdfsdf sdfsdfsdfsdfs"
| fields msg
| makemv delim="," msg
| mvexpand msg
| eval match=if(match(msg,"Allow.*(?=ProxyDeny|ProxyStrip)|^(?:(?!Allow).)*$"),1,0)
0 Karma

Influencer

You probably want to send to the parsingQueue, not the indexQueue too

0 Karma

Influencer

Thats because you're sending the lines that match this to the nullQueue, and sending all the stuff that doesn't match to the real queue.

You need to do

TRANSFORMS-noallow= setnull,setnoallow

and have

[setnoallow]
REGEX = regex_goes here
DEST_KEY = queue
FORMAT = parsingQueue
0 Karma

Engager

I still see Allow lines without ProxyDeny or ProxyStrip in the same line when i run a search in the index for last few minutes after restarting splunk.

Maybe did i made a mistake in the way i use transforms.conf ?

0 Karma

Engager

ok, i think i got it :

\sAllow\s.+msg="(?!(ProxyStrip|ProxyDeny))

0 Karma

Super Champion

Your regex seems to work fine in my tester. Are you seeing success in the search string, but not in your index?

0 Karma

Engager

Oh no, it still indexes all the "Allow" lines !

0 Karma