Splunk Dev

WineventLog are indexed late.

graju89
Path Finder

Hi all,

I am using splunk enterprise 7.1.4. I noticed some of the domain controllers logs(wineventlog) are indexed very late. The data is indexed 2.5 hrs late than the timestamp of the event. This is seen only on two domain controllers.

I need help or advise on this issue.

Thanks,

Tags (1)
0 Karma

lakshman239
Influencer

I assume the delays are seen from only Windows security events and not application or system events from those 2 domain controllers.

What's special/different on them compared to your other servers? Do you have a lot of security events on them? Is that in a network segment, where there can be delays? [ I assume the splunk conf/apps in all your AD servers are same]

0 Karma

graju89
Path Finder

@lakshman239 Yes, You are correct. But it delays for application logs as well. I am sure the events are higher than other servers. From splunk side I dont have any special changes for these servers.

0 Karma

lakshman239
Influencer

Does the delay go away after you re-boot the AD server? say for next few days?

0 Karma

graju89
Path Finder

I have not tried and can not do reboot. Those two AD servers are the main ones.

0 Karma

lakshman239
Influencer

Pls raise a case with splunk support

0 Karma

adonio
Ultra Champion

2.5 hours late (or early) might indicate India time or Iran time, only countries with 1/2 hour interval.
verify the cloak on your server as well as the time set for the user who looks at the data
you can also check the _indextime field and see if the event really "arrived" late, or your event time stamping / users set are off

hope it helps

0 Karma

graju89
Path Finder

Hi adonio,

I dont think it is timezone problem. The logs are indexed late not early. Most of the times it is late by 2.5hrs. Sometimes it indexes within 5 min. So I am guessing it is not time zone problem. Let me know if you have any other thoughts.

Thanks,

0 Karma

adonio
Ultra Champion

ill recommend to identify the latency patterns first:
... your search for windows ...| eval time=_time | eval itime=_indextime | eval latency=(itime - time) | stats count, avg(latency), min(latency), max(latency) by source

0 Karma

graju89
Path Finder

I tried that already. Latency is around 10000 sec(avg).

0 Karma

adonio
Ultra Champion

do you see latency from other sources?
did you measure network latency?
can you force a single event through the forwarder with add oneshot and measure results?

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...