The delay appears to be up to 15 minutes The delay is with the Symantec logs (sourcetype values: sep, sep:agt_system, sep:ids, and sep:behavior)
The delay is also inconsistent, meaning there are times where the logs are not delayed.
Has anyone seen this before?
If the "delay" is only 15 minutes, then I would guess that at least one of the servers that is generating the timestamps inside of the events has significant clock drift. You can see which one with this search:
index=* | eval timestamp=coalesce(timestamp,"OK") | eval date_zone=coalesce(date_zone,"none") | eval prev_sourcetype=if(sourcetype=$_sourcetype$,"none",_sourcetype) | dedup date_zone splunk_server index host sourcetype timestamp prev_sourcetype | eval lagSecs=_time-_indextime
Find the host
values that have bad lagSecs
times.
For clarification, Symantec logs are delayed getting into Splunk.