WineventLog are indexed late.

[graju89]

Hi all,

I am using splunk enterprise 7.1.4. I noticed some of the domain controllers logs(wineventlog) are indexed very late. The data is indexed 2.5 hrs late than the timestamp of the event. This is seen only on two domain controllers.

I need help or advise on this issue.

Thanks,

[lakshman239]

I assume the delays are seen from only Windows security events and not application or system events from those 2 domain controllers.

What's special/different on them compared to your other servers? Do you have a lot of security events on them? Is that in a network segment, where there can be delays? [ I assume the splunk conf/apps in all your AD servers are same]

[graju89]

@lakshman239 Yes, You are correct. But it delays for application logs as well. I am sure the events are higher than other servers. From splunk side I dont have any special changes for these servers.

[lakshman239]

Does the delay go away after you re-boot the AD server? say for next few days?

[graju89]

I have not tried and can not do reboot. Those two AD servers are the main ones.

[lakshman239]

Pls raise a case with splunk support

[adonio]

2.5 hours late (or early) might indicate India time or Iran time, only countries with 1/2 hour interval.
verify the cloak on your server as well as the time set for the user who looks at the data
you can also check the _indextime field and see if the event really "arrived" late, or your event time stamping / users set are off

hope it helps

[graju89]

Hi adonio,

I dont think it is timezone problem. The logs are indexed late not early. Most of the times it is late by 2.5hrs. Sometimes it indexes within 5 min. So I am guessing it is not time zone problem. Let me know if you have any other thoughts.

Thanks,

[adonio]

ill recommend to identify the latency patterns first:
... your search for windows ...| eval time=_time | eval itime=_indextime | eval latency=(itime - time) | stats count, avg(latency), min(latency), max(latency) by source

[graju89]

I tried that already. Latency is around 10000 sec(avg).

[adonio]

do you see latency from other sources?
did you measure network latency?
can you force a single event through the forwarder with add oneshot and measure results?