Splunk Search

Why does a real-time search with a small time range not return any results in Splunk 6.1.3?

nk-1
Path Finder

Sample Splunk Web search in Splunk 6.1.3 (Windows Server 2012):

host=MyHost level=INFO | stats count

always returns zero if I use Real Time 1-minute window.
If I change to Real Time 5-minute window, I get numbers that change every couple of seconds.

Why won't the 1-minute real-time window return results?

0 Karma
1 Solution

stephane_cyrill
Builder

Hi, When you simply do a ....|stats count ,splunk is doing statistics over all fields and that may take time so 1 minute window may be not be sufficient for that.

View solution in original post

nk-1
Path Finder

I'd just like to add a note that a reason why my 1-minute real-time window was not producing results when I went from indexing 1.5GB/day to 36GB/day was because the forwarders sending events to my indexers were, by default, configured to throttle after 256KB/second.
I changed maxKBps in limits.conf to zero in the forwarders, and the 1-minute real-time window displays updating counts now, without the need for clustering.

0 Karma

stephane_cyrill
Builder

Hi, When you simply do a ....|stats count ,splunk is doing statistics over all fields and that may take time so 1 minute window may be not be sufficient for that.

stephane_cyrill
Builder

Hi nk-1, feel free to vote and accept the answer. thanks

0 Karma

nk-1
Path Finder

Yes, this seems to make sense now.
I had radial gauges in my real-time dashboards that showed the count of incoming events in a 1-minute window.
It stopped working (always reporting zero) after I turned on DEBUG logging level on some application servers which increased incoming events from 1.5GB/day to about 36GB/day.

I might have to look at clustering Splunk to process things faster if I want the 1-min real-time reporting?

0 Karma
Get Updates on the Splunk Community!

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...

Adoption of Infrastructure Monitoring at Splunk

  Splunk's Growth Engineering team showcases one of their first Splunk product adoption-Splunk Infrastructure ...

Modern way of developing distributed application using OTel

Recently, I had the opportunity to work on a complex microservice using Spring boot and Quarkus to develop a ...