Developing for Splunk Enterprise

WineventLog are indexed late.

graju89
Path Finder

Hi all,

I am using splunk enterprise 7.1.4. I noticed some of the domain controllers logs(wineventlog) are indexed very late. The data is indexed 2.5 hrs late than the timestamp of the event. This is seen only on two domain controllers.

I need help or advise on this issue.

Thanks,

Tags (1)
0 Karma

lakshman239
SplunkTrust
SplunkTrust

I assume the delays are seen from only Windows security events and not application or system events from those 2 domain controllers.

What's special/different on them compared to your other servers? Do you have a lot of security events on them? Is that in a network segment, where there can be delays? [ I assume the splunk conf/apps in all your AD servers are same]

0 Karma

graju89
Path Finder

@lakshman239 Yes, You are correct. But it delays for application logs as well. I am sure the events are higher than other servers. From splunk side I dont have any special changes for these servers.

0 Karma

lakshman239
SplunkTrust
SplunkTrust

Does the delay go away after you re-boot the AD server? say for next few days?

0 Karma

graju89
Path Finder

I have not tried and can not do reboot. Those two AD servers are the main ones.

0 Karma

lakshman239
SplunkTrust
SplunkTrust

Pls raise a case with splunk support

0 Karma

adonio
SplunkTrust
SplunkTrust

2.5 hours late (or early) might indicate India time or Iran time, only countries with 1/2 hour interval.
verify the cloak on your server as well as the time set for the user who looks at the data
you can also check the _indextime field and see if the event really "arrived" late, or your event time stamping / users set are off

hope it helps

0 Karma

graju89
Path Finder

Hi adonio,

I dont think it is timezone problem. The logs are indexed late not early. Most of the times it is late by 2.5hrs. Sometimes it indexes within 5 min. So I am guessing it is not time zone problem. Let me know if you have any other thoughts.

Thanks,

0 Karma

adonio
SplunkTrust
SplunkTrust

ill recommend to identify the latency patterns first:
... your search for windows ...| eval time=_time | eval itime=_indextime | eval latency=(itime - time) | stats count, avg(latency), min(latency), max(latency) by source

0 Karma

graju89
Path Finder

I tried that already. Latency is around 10000 sec(avg).

0 Karma

adonio
SplunkTrust
SplunkTrust

do you see latency from other sources?
did you measure network latency?
can you force a single event through the forwarder with add oneshot and measure results?

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!