Splunk Search

Real-time views are hanging

Splunk Employee
Splunk Employee

I am attempting to use the real time view over time. It stops displaying events that are happening and hangs...the time it takes to hang after starting the view can range from minutes to hours. There is no indication of a problem. In fact everything appears normal, just no events are showing up. A non-real time search will show the missed events. The user restarts the real time search and it works for a while and then stops displaying events again...

Is there a setting to increase the expiration time for real time jobs?

Thank you!

Tags (2)
1 Solution

SplunkTrust
SplunkTrust

1) I've seen this happen on shorter timescales like 10 minutes, and what was causing it was

a) timechart is given an explicit span=5s argument,

b) the real-time time range is long enough that eventually there are 300 or 1000 rows in the output.

What happens then, is that it works at first, but since the FlashChart module has a default limit of 250 rows (which can be raised in the view configuration), once there are more than 250 rows, FlashChart only asks for only the topmost 250 rows in the results, which dont change anymore.

If this sounds like the same situation you're in, the solution is to either narrow the windowed real time range, dont use a hardcoded span or raise the hardcoded span to a bigger bucketsize, or raise the FlashChart limit from 250.

2) In general, if you see this happening in longer time scales like an hour, If there's no user interacting with the UI, the UI will eventually stop making requests once an hour passes. the updates will stop at that moment exactly and a while afterward, the user's sessions on splunkd and in splunkWeb will eventually time out.

We had the same annoying but simple problem on an internal demo that we kept up for a while.

One solution is to go to etc/system/local/web.conf, put a [settings] stanza in there if you dont have one already, and within that stanza set ui_inactivity_timeout to something higher than its default of 60 (minutes)

Note: this is an entirely different setting than the SplunkWeb "Session timeout" which is editable in the Manager section.

View solution in original post

SplunkTrust
SplunkTrust

1) I've seen this happen on shorter timescales like 10 minutes, and what was causing it was

a) timechart is given an explicit span=5s argument,

b) the real-time time range is long enough that eventually there are 300 or 1000 rows in the output.

What happens then, is that it works at first, but since the FlashChart module has a default limit of 250 rows (which can be raised in the view configuration), once there are more than 250 rows, FlashChart only asks for only the topmost 250 rows in the results, which dont change anymore.

If this sounds like the same situation you're in, the solution is to either narrow the windowed real time range, dont use a hardcoded span or raise the hardcoded span to a bigger bucketsize, or raise the FlashChart limit from 250.

2) In general, if you see this happening in longer time scales like an hour, If there's no user interacting with the UI, the UI will eventually stop making requests once an hour passes. the updates will stop at that moment exactly and a while afterward, the user's sessions on splunkd and in splunkWeb will eventually time out.

We had the same annoying but simple problem on an internal demo that we kept up for a while.

One solution is to go to etc/system/local/web.conf, put a [settings] stanza in there if you dont have one already, and within that stanza set ui_inactivity_timeout to something higher than its default of 60 (minutes)

Note: this is an entirely different setting than the SplunkWeb "Session timeout" which is editable in the Manager section.

View solution in original post