topic Re: WineventLog are indexed late. in Splunk Dev

WineventLog are indexed late.

graju89 — Mon, 28 Jan 2019 15:43:44 GMT

Hi all,

I am using splunk enterprise 7.1.4. I noticed some of the domain controllers logs(wineventlog) are indexed very late. The data is indexed 2.5 hrs late than the timestamp of the event. This is seen only on two domain controllers.

I need help or advise on this issue.

Thanks,

Re: WineventLog are indexed late.

adonio — Tue, 29 Jan 2019 00:09:56 GMT

2.5 hours late (or early) might indicate India time or Iran time, only countries with 1/2 hour interval.
verify the cloak on your server as well as the time set for the user who looks at the data
you can also check the _indextime field and see if the event really "arrived" late, or your event time stamping / users set are off

hope it helps

Re: WineventLog are indexed late.

graju89 — Fri, 01 Feb 2019 15:39:32 GMT

Hi adonio,

I dont think it is timezone problem. The logs are indexed late not early. Most of the times it is late by 2.5hrs. Sometimes it indexes within 5 min. So I am guessing it is not time zone problem. Let me know if you have any other thoughts.

Thanks,

Re: WineventLog are indexed late.

adonio — Fri, 01 Feb 2019 15:50:52 GMT

ill recommend to identify the latency patterns first:
... your search for windows ...| eval time=_time | eval itime=_indextime | eval latency=(itime - time) | stats count, avg(latency), min(latency), max(latency) by source

Re: WineventLog are indexed late.

graju89 — Fri, 01 Feb 2019 15:54:04 GMT

I tried that already. Latency is around 10000 sec(avg).

Re: WineventLog are indexed late.

adonio — Fri, 01 Feb 2019 16:17:45 GMT

do you see latency from other sources?
did you measure network latency?
can you force a single event through the forwarder with add oneshot and measure results?

Re: WineventLog are indexed late.

lakshman239 — Fri, 01 Feb 2019 16:51:41 GMT

I assume the delays are seen from only Windows security events and not application or system events from those 2 domain controllers.

What's special/different on them compared to your other servers? Do you have a lot of security events on them? Is that in a network segment, where there can be delays? [ I assume the splunk conf/apps in all your AD servers are same]

Re: WineventLog are indexed late.

graju89 — Fri, 01 Feb 2019 17:05:53 GMT

@lakshman239 Yes, You are correct. But it delays for application logs as well. I am sure the events are higher than other servers. From splunk side I dont have any special changes for these servers.

Re: WineventLog are indexed late.

lakshman239 — Fri, 01 Feb 2019 17:14:16 GMT

Does the delay go away after you re-boot the AD server? say for next few days?

Re: WineventLog are indexed late.

graju89 — Fri, 01 Feb 2019 17:16:53 GMT

I have not tried and can not do reboot. Those two AD servers are the main ones.

Re: WineventLog are indexed late.

lakshman239 — Mon, 04 Feb 2019 15:08:25 GMT

Pls raise a case with splunk support