topic Re: Date latency in Monitoring Splunk

Date latency

uagraw01 — Fri, 19 Jun 2020 17:58:19 GMT

I am receiving the logs from the forwarders and can see latency between index time and event time. We have difference between index time and event time is about 15 to 16 hours on more than 300 forwarders. How can i fix this issue?

Re: Date latency

richgalloway — Fri, 19 Jun 2020 18:27:21 GMT

That's not (usually) a simple fix. There are a variety of causes and finding the root cause will likely require intimate knowledge of your environment.

Some things to check include:

All servers are using NTP (or an equivalent time-sync service)
Time zones are set properly on each server
Event timestamps include a time zone indication or inputs.conf contains the TZ attribute
Props.conf has TIME_FORMAT attributes that correctly extract the time zone from event timestamps
All Splunk forwarders are always running
Any intermediate servers or processes are always running
Events are not cached by the generating server/process before they are sent to Splunk

Re: Date latency

uagraw01 — Tue, 23 Jun 2020 05:53:04 GMT

@richgalloway Is DATETIME_CONFIG = CURRENT will work ?

Re: Date latency

richgalloway — Tue, 23 Jun 2020 12:29:18 GMT

It will "work" in that it will assign the current time to each event that arrives. It masks the latency problem. It makes old events look like new events and may throw off your reports.

Re: Date latency

uagraw01 — Tue, 23 Jun 2020 13:15:33 GMT

@richgalloway Any other solution you can suggest to me. Because our thruput limit is set to 1024kb and that is fine . Any major issue we can fix this permanently.

Re: Date latency

richgalloway — Tue, 23 Jun 2020 13:41:24 GMT

I offered 7 possible solutions in my first reply. Have you checked them?