Splunk Search

Real-time views are hanging

Jaci
Splunk Employee
Splunk Employee

I am attempting to use the real time view over time. It stops displaying events that are happening and hangs...the time it takes to hang after starting the view can range from minutes to hours. There is no indication of a problem. In fact everything appears normal, just no events are showing up. A non-real time search will show the missed events. The user restarts the real time search and it works for a while and then stops displaying events again...

Is there a setting to increase the expiration time for real time jobs?

Thank you!

Tags (2)
1 Solution

sideview
SplunkTrust
SplunkTrust

1) I've seen this happen on shorter timescales like 10 minutes, and what was causing it was

a) timechart is given an explicit span=5s argument,

b) the real-time time range is long enough that eventually there are 300 or 1000 rows in the output.

What happens then, is that it works at first, but since the FlashChart module has a default limit of 250 rows (which can be raised in the view configuration), once there are more than 250 rows, FlashChart only asks for only the topmost 250 rows in the results, which dont change anymore.

If this sounds like the same situation you're in, the solution is to either narrow the windowed real time range, dont use a hardcoded span or raise the hardcoded span to a bigger bucketsize, or raise the FlashChart limit from 250.

2) In general, if you see this happening in longer time scales like an hour, If there's no user interacting with the UI, the UI will eventually stop making requests once an hour passes. the updates will stop at that moment exactly and a while afterward, the user's sessions on splunkd and in splunkWeb will eventually time out.

We had the same annoying but simple problem on an internal demo that we kept up for a while.

One solution is to go to etc/system/local/web.conf, put a [settings] stanza in there if you dont have one already, and within that stanza set ui_inactivity_timeout to something higher than its default of 60 (minutes)

Note: this is an entirely different setting than the SplunkWeb "Session timeout" which is editable in the Manager section.

View solution in original post

sideview
SplunkTrust
SplunkTrust

1) I've seen this happen on shorter timescales like 10 minutes, and what was causing it was

a) timechart is given an explicit span=5s argument,

b) the real-time time range is long enough that eventually there are 300 or 1000 rows in the output.

What happens then, is that it works at first, but since the FlashChart module has a default limit of 250 rows (which can be raised in the view configuration), once there are more than 250 rows, FlashChart only asks for only the topmost 250 rows in the results, which dont change anymore.

If this sounds like the same situation you're in, the solution is to either narrow the windowed real time range, dont use a hardcoded span or raise the hardcoded span to a bigger bucketsize, or raise the FlashChart limit from 250.

2) In general, if you see this happening in longer time scales like an hour, If there's no user interacting with the UI, the UI will eventually stop making requests once an hour passes. the updates will stop at that moment exactly and a while afterward, the user's sessions on splunkd and in splunkWeb will eventually time out.

We had the same annoying but simple problem on an internal demo that we kept up for a while.

One solution is to go to etc/system/local/web.conf, put a [settings] stanza in there if you dont have one already, and within that stanza set ui_inactivity_timeout to something higher than its default of 60 (minutes)

Note: this is an entirely different setting than the SplunkWeb "Session timeout" which is editable in the Manager section.

Get Updates on the Splunk Community!

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...