Splunk Search

Why enabling real time search does not display data immediately?

Communicator

I am receiving events every 15 seconds. But when I enable real time search in default splunk search app for query sourcetype="mysource" with 1 minute window, It does not display the events arriving after enabling this search. If I see the timeline, it shows the current time is proceeding but no event appears. I see the the count against the scanned events but matching events remains 0. It takes a while (almost 3-4 minutes) after which the events starts appearing as expected.

If I enable real-time for 'All Time' then everything works as expected.

Is there anything which I am missing about the real-time search.

Tags (2)
0 Karma
1 Solution

Splunk Employee
Splunk Employee

It sounds like 1 of 2 things may be happening.

  1. Events could be arriving late. If events take more than 60s to reach the indexer then they wont meet the time bounds of rt-1m to rt (1 minute window) and won't be displayed. If you are receiving events every 15s then this seems less likely.
  2. You have some clock skew going on. If events have older time stamps they effectively reduce the 60s window. If all events are time stamped 1 minute in the past then the window is effectively zero.

Based on the comment about waiting 3-4 minutes for an event, it sounds like your indexer may be ahead 4 minutes relative to the data source.

View solution in original post

Splunk Employee
Splunk Employee

It sounds like 1 of 2 things may be happening.

  1. Events could be arriving late. If events take more than 60s to reach the indexer then they wont meet the time bounds of rt-1m to rt (1 minute window) and won't be displayed. If you are receiving events every 15s then this seems less likely.
  2. You have some clock skew going on. If events have older time stamps they effectively reduce the 60s window. If all events are time stamped 1 minute in the past then the window is effectively zero.

Based on the comment about waiting 3-4 minutes for an event, it sounds like your indexer may be ahead 4 minutes relative to the data source.

View solution in original post

Communicator

Thanks. It was the second reason.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!