Splunk Search

Why does my dashboard panel search crash Splunk if the search time frame is less than 15 minutes?

daishih
Path Finder

I created a two panel dashboard I want to use to see "block" OR "deny" firewall records from three of our security devices. One panel searches for these by using the IP address that is input into a token field. I can run Real-time 30 second window searches using this panel. The search string for this fully functional panel is:

blocked OR deny  | search src="$IP$" OR src_ip="$IP$"  | eval "Source IP" = coalesce(src, src_ip, src_host)  | eval "Destination IP" = coalesce(dst, dst_ip, dest_ip)  | eval "Destination Port" = coalesce(dst_port, dest_port)  | eval "Application" = coalesce(service, appname, application)  | eval "Destination Zone" = coalesce(dst_zone, dest_zone) | table _time, sourcetype,  "Source IP", "Destination IP", "Destination Port", url, "Application", category, src_zone, "Destination Zone", policy_id, action

The problem is with the second panel that searches using the username is input into a token field. This search contains a sub string and will only run using a 15 minute or longer time frame. If I try a time frame shorter than 15 minutes, such as Real-time 30 second or Real-time one minute, the search bar moves a few centimeters then stops and shows "Waiting for data..." after about a few minutes the webpage crashes. I've tried this in Chrome, IE and Firefox, but the result is the same. The search string used for this panel is:

blocked OR deny [search sourcetype=pan:traffic  | where user="mydom\\$userName$"  | rename src_ip AS src | fields src]  | eval "Source IP" = coalesce(src, src_ip, src_host)  | eval "Destination IP" = coalesce(dst, dst_ip, dest_ip)  | eval "Destination Port" = coalesce(dst_port, dest_port)  | eval "Application" = coalesce(service, appname, application)  | eval "Destination Zone" = coalesce(dst_zone, dest_zone) | table _time, sourcetype,  "Source IP", "Destination IP", "Destination Port", url, "Application", category, src_zone, "Destination Zone", policy_id, action

Is there a reason the panel that uses the sub string search is doing this? Ideally I'd like that panel to work with every time frame including Real-time 30 second window.

Thank you,

0 Karma
1 Solution

MuS
SplunkTrust
SplunkTrust

Hi daishih,

The problem is your subsearch; the subsearch result is used as a parameter for the main search and it is run as first search in the stream. But since a real-time search does not finish, it cannot be used as a subsearch.
Change your search so it does not use a subsearch, plenty of hints available :

https://answers.splunk.com/answers/129424/how-to-compare-fields-over-multiple-sourcetypes-without-jo...
http://wiki.splunk.com/Virtual_.conf March session 2016

Another option would be to change the real time search to be a saved search using earliest=-1min time range. There are multiple reasons why you can run into trouble using real time searches (time stamping or late arriving events for example).

Hope this helps ...

cheers, MuS

View solution in original post

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi daishih,

The problem is your subsearch; the subsearch result is used as a parameter for the main search and it is run as first search in the stream. But since a real-time search does not finish, it cannot be used as a subsearch.
Change your search so it does not use a subsearch, plenty of hints available :

https://answers.splunk.com/answers/129424/how-to-compare-fields-over-multiple-sourcetypes-without-jo...
http://wiki.splunk.com/Virtual_.conf March session 2016

Another option would be to change the real time search to be a saved search using earliest=-1min time range. There are multiple reasons why you can run into trouble using real time searches (time stamping or late arriving events for example).

Hope this helps ...

cheers, MuS

0 Karma

daishih
Path Finder

Do you know if it is possible to pass the source IP obtained from the username search into the IP field for the IP search? This would save us from having to construct a complex search since the current IP search works fine with real-time time frames.

0 Karma

MuS
SplunkTrust
SplunkTrust

Have you tried something like this:

 blocked OR deny AND sourcetype=pan:traffic AND user="mydom\\$userName$" | rename src_ip AS src | eval BadGuy=if(sourcetype=pan:traffic, 1, null()) | streamstats anything you need

further down the search pipe you can use a where BadGuy=1 and do more Splunk-Fu afterwards.
Again check out the provided links which provide multiple excellent examples to do it.

cheers, MuS

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...