REGEX to discard specific event containing " Allow " but not them followad by 2 specific strings ?

[fgilain]

Hello,

i need to find the REGEX to allow me to filter what splunk will index.

As it is firewall Logs, it gererates too much volume, so i want to remove events containing the following pattern :

" Allow " (except if there is "ProxyStrip" or "ProxyDeny" somewhere after in the line)

example :

Would match :
Sep 30 16:43:07 host=firewall-01 program=XXX0292D4BCD PID= facility=local1 level=info : firewall-01 http-proxy[3949]: Allow 3-LAN 2-EXTERNAL tcp 192.168.XXX.2 37.XXX.XXX.66 62745 80 msg="ProxyAllow: HTTP Header content type match" proxy_act="HTTP-Client.WebBlocker" rule_name="image/*" content_type="image/gif" (Out_Users_HTTP-rule)

Would not match :

Sep 30 16:43:07 host=firewall-01 program=XXX0292D4BCD PID= facility=local1 level=info : firewall-01 http-proxy[3949]: Allow 3-LAN 2-EXTERNAL tcp 192.168.XXX.2 37.XXX.XXX.66 62745 80 msg="ProxyStrip: HTTP Header content type match" proxy_act="HTTP-Client.WebBlocker" rule_name="image/*" content_type="image/gif" (Out_Users_HTTP-rule)

Sep 30 16:43:07 host=firewall-01 program=XXX0292D4BCD PID= facility=local1 level=info : firewall-01 http-proxy[3949]: Allow 3-LAN 2-EXTERNAL tcp 192.168.XXX.2 37.XXX.XXX.66 62745 80 msg="ProxyDeny: HTTP Header content type match" proxyact="HTTP-Client.WebBlocker" rulename="image/*" contenttype="image/gif" (OutUsers_HTTP-rule)

Any idea ?

Thanks for your help

Florent

[andreas]

try this as REGEX for the nullQueue:

\sAllow\s(?!.*(?:ProxyStrip|ProxyDeny))

or if you also want to include the ' msg="' part:

\sAllow\s(?!.*\smsg="(?:ProxyStrip|ProxyDeny))

[fgilain]

i want to index only "Deny" + "Allow" (for "Allow" : only if followed by ProxyStrip or ProxyDeny only).

[jonuwz]

This is confusing.

You want to filter out if the regex matches.

So what you're saying is you want lines line Allow ProxyStrip and Allow ProxyDeny to be indexed, but ProxyAllow to be discarded ?

[fgilain]

In transforms.conf (local one) :

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = Deny
DEST_KEY = queue
FORMAT = indexQueue

[setnoallow]
REGEX = Allow.(?=ProxyDeny|ProxyStrip)|^(?:(?!Allow).)$
DEST_KEY = queue
FORMAT = nullQueue

In props.conf (local one) :

[source::/Logs/Firewall-logs/firewall-01/firewall-01_local0.log]
TRANSFORMS-noallow= setnoallow

[source::/Logs/Firewall-logs/firewall-01/firewall-01_local1.log]
TRANSFORMS-noallow= setnoallow

[jonuwz]

"Allow.*(?=ProxyDeny|ProxyStrip)|^(?:(?!Allow).)*$"

Match "Allow followed by ProxyDeny or ProxyStrip" or "Any line without Allow"

Test :

| stats count 
| eval msg="Allow skldfjlksdjflksdjflksjdf ProxyDeny,Allow skldfjlksdjflksdjflksjdf ProxyAccept,Deny zasdfasdfsdf sdfsdfsdfsdfs"
| fields msg
| makemv delim="," msg
| mvexpand msg
| eval match=if(match(msg,"Allow.*(?=ProxyDeny|ProxyStrip)|^(?:(?!Allow).)*$"),1,0)

[jonuwz]

You probably want to send to the parsingQueue, not the indexQueue too

[jonuwz]

Thats because you're sending the lines that match this to the nullQueue, and sending all the stuff that doesn't match to the real queue.

You need to do

TRANSFORMS-noallow= setnull,setnoallow

and have

[setnoallow]
REGEX = regex_goes here
DEST_KEY = queue
FORMAT = parsingQueue

[fgilain]

I still see Allow lines without ProxyDeny or ProxyStrip in the same line when i run a search in the index for last few minutes after restarting splunk.

Maybe did i made a mistake in the way i use transforms.conf ?

[fgilain]

ok, i think i got it :

\sAllow\s.+msg="(?!(ProxyStrip|ProxyDeny))

[lukejadamec]

Your regex seems to work fine in my tester. Are you seeing success in the search string, but not in your index?

[fgilain]

Oh no, it still indexes all the "Allow" lines !