Splunk Search

Search using join returning incoherent results

Explorer

Hello,

I am getting different results running the same search over the same interval of time.
The search is:

sourcetype=incapsula (siteid="*" OR sourceServiceName="*") Action=REQ_BLOCKED* CEF_Severity>=0 src=* | transaction src | join src  [search sourcetype=cisco:asa message_id=302013 OR message_id=302015 OR message_id=302020  Cisco_ASA_action=allowed src=* | transaction src] | table src, CEF_Name, Action, request, Cisco_ASA_action, dest_ip, dest_port

As I am running a subsearch, I suspect that may be a timeout problem as I know that by default the time limit for subsearches is 60 seconds and this subsearch is taking around 69 seconds to run.
If this is the problem, can you help me making the search more performant?

Another strange thing is that when I "inspect" de job, I can see that the normalized search includes some conditions that I didn't include in the search.

This is a sample of the "Normalized search" on the job inspector:

Normalized search:

litsearch ( sourcetype="cisco:asa" OR sourcetype=ciscoasa ) messageid=302013 OR messageid=302015 OR messageid=302020 ( ( ( ( sourcetype="cisco:asa" OR sourcetype=ciscoasa ) ) AND ( ( ( CiscoASAvendoraction=built ) ) OR ( ( CiscoASAvendoraction=permitted ) ) OR ( ( CiscoASAvendoraction="Pre-allocated" ) ) OR ( ( CiscoASAvendoraction=whitelisted ) ) OR ( ( CiscoASAvendoraction=Rebuilt ) ) OR ( ( CiscoASAvendoraction="permitted tcp" ) ) OR ( ( CiscoASAvendoraction="permitted udp" ) ) OR ( ( CiscoASAvendoraction="permitted icmp" ) ) OR ( ( CiscoASAvendoraction="access requested" ) ) OR ( ( CiscoASAvendoraction="access permitted" ) ) OR ( ( CiscoASAvendoraction=accessed ) ) ) ) OR ( ( ( sourcetype="cisco:fwsm" OR sourcetype=ciscofwsm ) ) AND ( ( ( CiscoASAvendoraction=built ) ) OR ( ( CiscoASAvendoraction=permitted ) ) OR ( ( CiscoASAvendoraction="Pre-allocated" ) ) OR ( ( CiscoASAvendoraction=whitelisted ) ) OR ( ( CiscoASAvendoraction=Rebuilt ) ) OR ( ( CiscoASAvendoraction="permitted tcp" ) ) OR ( ( CiscoASAvendoraction="permitted udp" ) ) OR ( ( CiscoASAvendoraction="permitted icmp" ) ) OR ( ( CiscoASAvendoraction="access requested" ) ) OR ( ( CiscoASAvendoraction="access permitted" ) ) OR ( ( CiscoASAvendoraction=accessed ) ) ) ) OR ( ( ( sourcetype="cisco:pix" OR sourcetype=ciscopix ) ) AND ( ( ( CiscoASAvendoraction=built ) ) OR ( ( CiscoASAvendoraction=permitted ) ) OR ( ( CiscoASAvendoraction="Pre-allocated" ) ) OR ( ( CiscoASAvendoraction=whitelisted ) ) OR ( ( CiscoASAvendoraction=Rebuilt ) ) OR ( ( CiscoASAvendoraction="permitted tcp" ) ) OR ( ( CiscoASAvendoraction="permitted udp" ) ) OR ( ( CiscoASAvendoraction="permitted icmp" ) ) OR ( ( CiscoASAvendoraction="access requested" ) ) OR ( ( CiscoASAvendoraction="access permitted" ) ) OR ( ( CiscoASAvendoraction=accessed ) ) ) ) ) OR ( CiscoASAaction=allowed ) ( ( ( ( sourcetype="bluecoat:proxysg:access:file" ) AND ( ( cip="*" ) ) ) OR ( ( ( sourcetype="bluecoat:proxysg:access:syslog" OR sourcetype=bluecoat ) ) AND ( ( cip="" ) ) ) OR ( ( sourcetype="cisco:ios" ) AND ( ( src_ip="" ) ) ) OR ( ( sourcetype=cpu ) AND ( ( host="" ) ) ) OR ( ( sourcetype=df ) AND ( ( host="" ) ) ) OR ( ( sourcetype=iis ) AND ( ( cip="*" ) ) ) OR ( ( sourcetype="mcafee:epo" ) AND ( ( srcip="" ) ) ) OR ( ( sourcetype="mcafee:ids" ) AND ( ( SIP="" ) ) ) OR ( ( sourcetype="oracle:listener:text" ) AND ( ( CLIENTIP="" ) ) ) OR ( ( sourcetype="oracle:listener:xml" ) AND ( ( CLIENTIP="" ) ) ) OR ( ( sourcetype=ps ) AND ( ( host="" ) ) ) OR ( ( source="" ) AND ( ( host="" ) ) ) OR ( ( source="" ) AND ( ( host="" ) ) ) OR ( ( source="Perfmon" ) AND ( ( host="" ) ) ) OR ( ( source="WMI" ) AND ( ( host="" ) ) ) OR ( ( sourcetype=vmstat ) AND ( ( host="" ) ) ) ) OR ( ( src="" OR sourcetype=aixsecure OR sourcetype=linuxsecure OR sourcetype=osxsecure OR sourcetype=syslog ) ) OR ( ( sourcetype="cisco:asa" OR sourcetype=ciscoasa ) ) OR ( ( sourcetype="cisco:fwsm" OR sourcetype=ciscofwsm ) ) OR ( ( sourcetype="cisco:pix" OR sourcetype=ciscopix ) ) OR ( sourcetype="oracle:audit:text" ) OR ( sourcetype="oracle:audit:xml" ) ) | litsearch sourcetype=cisco:asa messageid=302013 OR messageid=302015 OR messageid=302020 CiscoASA_action=allowed src= | fields keepcolorder=t "*" "bkt" "cd" "si" "txnendswith" "txnfiltermatch" "txnstartswith" "host" "index" "linecount" "source" "sourcetype" "splunk_server" "src" | pretransaction src

Why is this happening?

As this is a client's enviroment, I don't have access to the filesystem to check the .conf files.

Can you help me with this issue?

Thank you in advance.
Regards

0 Karma
1 Solution

SplunkTrust
SplunkTrust

Give this a try

Updated

(sourcetype=incapsula (siteid="*" OR sourceServiceName="*") Action=REQ_BLOCKED* CEF_Severity>=0 src=*) OR ( sourcetype=cisco:asa message_id=302013 OR message_id=302015 OR message_id=302020  Cisco_ASA_action=allowed src=* ) 
| fields sourcetype src, CEF_Name, Action, request, Cisco_ASA_action, dest_ip, dest_port
| stats dc(sourcetype) as sts values(CEF_Name) as CEF_Name values(Action) as Action values(request) as request values(Cisco_ASA_action) as Cisco_ASA_action values(dest_ip) as dest_ip values(dest_port) as dest_port by src 
| where sts=2 | fields - sts

View solution in original post

SplunkTrust
SplunkTrust

Give this a try

Updated

(sourcetype=incapsula (siteid="*" OR sourceServiceName="*") Action=REQ_BLOCKED* CEF_Severity>=0 src=*) OR ( sourcetype=cisco:asa message_id=302013 OR message_id=302015 OR message_id=302020  Cisco_ASA_action=allowed src=* ) 
| fields sourcetype src, CEF_Name, Action, request, Cisco_ASA_action, dest_ip, dest_port
| stats dc(sourcetype) as sts values(CEF_Name) as CEF_Name values(Action) as Action values(request) as request values(Cisco_ASA_action) as Cisco_ASA_action values(dest_ip) as dest_ip values(dest_port) as dest_port by src 
| where sts=2 | fields - sts

View solution in original post

Explorer

Thank for your response but we need to get the events that have a src that appears on both sourcetypes.
With your search, we get all the events not only the ones that have the same src on both sourcetypes.

Thank's again.

0 Karma

SplunkTrust
SplunkTrust

Almost forgot about that. Try the updated answer now.

0 Karma

Explorer

Thanks!!! It works perfectly.

0 Karma

SplunkTrust
SplunkTrust

You're using two of the most expensive command, transaction and join. can you list down the field name in each sourcetype that you want to see in the final output?

0 Karma

Explorer

Thanks for your reply,

for the sourcetype= incapsula:
src, CEF_Name, Action, request

for the cisco sourcetype:
CiscoASAaction, destip, destport

Thanks!!

0 Karma