Splunk Search

Why enabling real time search does not display data immediately?

asingla
Communicator

I am receiving events every 15 seconds. But when I enable real time search in default splunk search app for query sourcetype="mysource" with 1 minute window, It does not display the events arriving after enabling this search. If I see the timeline, it shows the current time is proceeding but no event appears. I see the the count against the scanned events but matching events remains 0. It takes a while (almost 3-4 minutes) after which the events starts appearing as expected.

If I enable real-time for 'All Time' then everything works as expected.

Is there anything which I am missing about the real-time search.

Tags (2)
0 Karma
1 Solution

jflomenberg
Splunk Employee
Splunk Employee

It sounds like 1 of 2 things may be happening.

  1. Events could be arriving late. If events take more than 60s to reach the indexer then they wont meet the time bounds of rt-1m to rt (1 minute window) and won't be displayed. If you are receiving events every 15s then this seems less likely.
  2. You have some clock skew going on. If events have older time stamps they effectively reduce the 60s window. If all events are time stamped 1 minute in the past then the window is effectively zero.

Based on the comment about waiting 3-4 minutes for an event, it sounds like your indexer may be ahead 4 minutes relative to the data source.

View solution in original post

jflomenberg
Splunk Employee
Splunk Employee

It sounds like 1 of 2 things may be happening.

  1. Events could be arriving late. If events take more than 60s to reach the indexer then they wont meet the time bounds of rt-1m to rt (1 minute window) and won't be displayed. If you are receiving events every 15s then this seems less likely.
  2. You have some clock skew going on. If events have older time stamps they effectively reduce the 60s window. If all events are time stamped 1 minute in the past then the window is effectively zero.

Based on the comment about waiting 3-4 minutes for an event, it sounds like your indexer may be ahead 4 minutes relative to the data source.

asingla
Communicator

Thanks. It was the second reason.

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...