Splunk Search

Why enabling real time search does not display data immediately?

asingla
Communicator

I am receiving events every 15 seconds. But when I enable real time search in default splunk search app for query sourcetype="mysource" with 1 minute window, It does not display the events arriving after enabling this search. If I see the timeline, it shows the current time is proceeding but no event appears. I see the the count against the scanned events but matching events remains 0. It takes a while (almost 3-4 minutes) after which the events starts appearing as expected.

If I enable real-time for 'All Time' then everything works as expected.

Is there anything which I am missing about the real-time search.

Tags (2)
0 Karma
1 Solution

jflomenberg
Splunk Employee
Splunk Employee

It sounds like 1 of 2 things may be happening.

  1. Events could be arriving late. If events take more than 60s to reach the indexer then they wont meet the time bounds of rt-1m to rt (1 minute window) and won't be displayed. If you are receiving events every 15s then this seems less likely.
  2. You have some clock skew going on. If events have older time stamps they effectively reduce the 60s window. If all events are time stamped 1 minute in the past then the window is effectively zero.

Based on the comment about waiting 3-4 minutes for an event, it sounds like your indexer may be ahead 4 minutes relative to the data source.

View solution in original post

jflomenberg
Splunk Employee
Splunk Employee

It sounds like 1 of 2 things may be happening.

  1. Events could be arriving late. If events take more than 60s to reach the indexer then they wont meet the time bounds of rt-1m to rt (1 minute window) and won't be displayed. If you are receiving events every 15s then this seems less likely.
  2. You have some clock skew going on. If events have older time stamps they effectively reduce the 60s window. If all events are time stamped 1 minute in the past then the window is effectively zero.

Based on the comment about waiting 3-4 minutes for an event, it sounds like your indexer may be ahead 4 minutes relative to the data source.

asingla
Communicator

Thanks. It was the second reason.

0 Karma
Get Updates on the Splunk Community!

Announcing the Expansion of the Splunk Academic Alliance Program

The Splunk Community is more than just an online forum — it’s a network of passionate users, administrators, ...

Learn Splunk Insider Insights, Do More With Gen AI, & Find 20+ New Use Cases You Can ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Buttercup Games: Further Dashboarding Techniques (Part 7)

This series of blogs assumes you have already completed the Splunk Enterprise Search Tutorial as it uses the ...