Why does my dashboard panel search crash Splunk if the search time frame is less than 15 minutes?

[daishih]

I created a two panel dashboard I want to use to see "block" OR "deny" firewall records from three of our security devices. One panel searches for these by using the IP address that is input into a token field. I can run Real-time 30 second window searches using this panel. The search string for this fully functional panel is:

blocked OR deny  | search src="$IP$" OR src_ip="$IP$"  | eval "Source IP" = coalesce(src, src_ip, src_host)  | eval "Destination IP" = coalesce(dst, dst_ip, dest_ip)  | eval "Destination Port" = coalesce(dst_port, dest_port)  | eval "Application" = coalesce(service, appname, application)  | eval "Destination Zone" = coalesce(dst_zone, dest_zone) | table _time, sourcetype,  "Source IP", "Destination IP", "Destination Port", url, "Application", category, src_zone, "Destination Zone", policy_id, action

The problem is with the second panel that searches using the username is input into a token field. This search contains a sub string and will only run using a 15 minute or longer time frame. If I try a time frame shorter than 15 minutes, such as Real-time 30 second or Real-time one minute, the search bar moves a few centimeters then stops and shows "Waiting for data..." after about a few minutes the webpage crashes. I've tried this in Chrome, IE and Firefox, but the result is the same. The search string used for this panel is:

blocked OR deny [search sourcetype=pan:traffic  | where user="mydom\\$userName$"  | rename src_ip AS src | fields src]  | eval "Source IP" = coalesce(src, src_ip, src_host)  | eval "Destination IP" = coalesce(dst, dst_ip, dest_ip)  | eval "Destination Port" = coalesce(dst_port, dest_port)  | eval "Application" = coalesce(service, appname, application)  | eval "Destination Zone" = coalesce(dst_zone, dest_zone) | table _time, sourcetype,  "Source IP", "Destination IP", "Destination Port", url, "Application", category, src_zone, "Destination Zone", policy_id, action

Is there a reason the panel that uses the sub string search is doing this? Ideally I'd like that panel to work with every time frame including Real-time 30 second window.

Thank you,

[MuS · Accepted solution]

Hi daishih,

The problem is your subsearch; the subsearch result is used as a parameter for the main search and it is run as first search in the stream. But since a real-time search does not finish, it cannot be used as a subsearch.
Change your search so it does not use a subsearch, plenty of hints available :

https://answers.splunk.com/answers/129424/how-to-compare-fields-over-multiple-sourcetypes-without-jo...
http://wiki.splunk.com/Virtual_.conf March session 2016

Another option would be to change the real time search to be a saved search using earliest=-1min time range. There are multiple reasons why you can run into trouble using real time searches (time stamping or late arriving events for example).

Hope this helps ...

cheers, MuS

[daishih]

Do you know if it is possible to pass the source IP obtained from the username search into the IP field for the IP search? This would save us from having to construct a complex search since the current IP search works fine with real-time time frames.

[MuS]

Have you tried something like this:

 blocked OR deny AND sourcetype=pan:traffic AND user="mydom\\$userName$" | rename src_ip AS src | eval BadGuy=if(sourcetype=pan:traffic, 1, null()) | streamstats anything you need

further down the search pipe you can use a where BadGuy=1 and do more Splunk-Fu afterwards.
Again check out the provided links which provide multiple excellent examples to do it.

cheers, MuS