I am using splunk enterprise 7.1.4. I noticed some of the domain controllers logs(wineventlog) are indexed very late. The data is indexed 2.5 hrs late than the timestamp of the event. This is seen only on two domain controllers.
I need help or advise on this issue.
I assume the delays are seen from only Windows security events and not application or system events from those 2 domain controllers.
What's special/different on them compared to your other servers? Do you have a lot of security events on them? Is that in a network segment, where there can be delays? [ I assume the splunk conf/apps in all your AD servers are same]
@lakshman239 Yes, You are correct. But it delays for application logs as well. I am sure the events are higher than other servers. From splunk side I dont have any special changes for these servers.
2.5 hours late (or early) might indicate India time or Iran time, only countries with 1/2 hour interval.
verify the cloak on your server as well as the time set for the user who looks at the data
you can also check the
_indextime field and see if the event really "arrived" late, or your event time stamping / users set are off
hope it helps
I dont think it is timezone problem. The logs are indexed late not early. Most of the times it is late by 2.5hrs. Sometimes it indexes within 5 min. So I am guessing it is not time zone problem. Let me know if you have any other thoughts.
ill recommend to identify the latency patterns first:
... your search for windows ...| eval time=_time | eval itime=_indextime | eval latency=(itime - time) | stats count, avg(latency), min(latency), max(latency) by source