topic Re: Logs delay due to timezone difference in the UF server in Getting Data In

How can I fix logs delay due to timezone difference in the UF server?

sekhar463 — Mon, 05 Jun 2023 21:07:42 GMT

Hi All i have a log source in the server timezone is in CST and logs are coming into the server as UTC time zone logs.

so while ingesting logs splunk was ingesting based on CST which is 5 hours before logs.

i have added props config for UTC but still getting the issue.

please let us know what needs to be done

Re: Logs delay due to timezone difference in the UF server

isoutamo — Mon, 05 Jun 2023 16:05:55 GMT

Splunk use internally UTC for all times. When you are ingesting event splunk automatically change it's time to UTC based on your source system timezone and if it cannot determine it when events are collected then it use indexers time zone. When you are searching then it's use your web browser's time zone unless you haven't separately defined it in your Splunk's User preferences.

If your serves (UF etc) haven't correct TZ information and time is not in sync, you should fix this. Another option is use conf files on UF and Indexer/HF side to fix this issue.

More about timezone handling in splunk https://docs.splunk.com/Documentation/Splunk/latest/Search/Abouttimezones

r. Ismo

Re: Logs delay due to timezone difference in the UF server

diogofgm — Mon, 05 Jun 2023 16:08:34 GMT

You can create a stanza in props.conf with:

[host::<host>] TZ=<TZ>

Re: Logs delay due to timezone difference in the UF server

PickleRick — Mon, 05 Jun 2023 20:38:33 GMT

There can be many things that went wrong and you're not providing too much information 😉

@isoutamowas a tiny bit imprecise saying that splunk uses UTC internally. Actually splunk internally stores _time as a "unix timestamp" which just points to a given point in time and can be rendered to a human-readable time string in any timezone. In order to do so Splunk has to parse the human readable (or not) timestamp from the event.

So the problem with event's time can be in three separate places (in each of them or even if all three of them at the same time).

1. The source logs can have wrong timestamp (happens with badly configured source - for example, if someone configured a device to store local time but report it as UTC)

2. The log is wrongly parsed due to misconfiguration (or if there was no explicit configuration at all and Splunk's "guess" as to the timestamp format turns out to be wrong).

2a. There is no timezone information present in the event and splunk applies wrong TZ to the timesamp.

3. The user's timezone is misconfigured in preferences and the timestamp is wrongly rendered in the webui.

So depending on what is the problem, there should be proper fix

1. Configure source properly

2. Apply time extraction configuration on the indexers (or HFs if you have them) - in a typical example

2a. Add TZ definition for a given sourcetype/source/host - this can be a bit more tricky than what @diogofgm showed if you have multiple event sources from one host - each with different TZ

3. Change your user's preferences.

Re: Logs delay due to timezone difference in the UF server

sekhar463 — Tue, 06 Jun 2023 08:49:55 GMT

i have already added props file with sourcetype but didnt fixed.

[netapp:cloudsecure:agentlog]
TZ = UTC

Re: Logs delay due to timezone difference in the UF server

PickleRick — Tue, 06 Jun 2023 16:01:03 GMT

But on which component did you put that setting?

Re: Logs delay due to timezone difference in the UF server

sekhar463 — Tue, 06 Jun 2023 16:59:15 GMT

i have added props.conf in the Deployment server for the specific app to update in the UF

Re: Logs delay due to timezone difference in the UF server

PickleRick — Tue, 06 Jun 2023 19:46:06 GMT

And that's why it's not working 🙂

UF is not a parsing component. UF just picks up data, splits it up into chunks and sends to indexer(s) or intermediate forwarder(s). You need that setting on your parsing components (indexers or heavy forwarders) in the path of your events.

Re: Logs delay due to timezone difference in the UF server

sekhar463 — Wed, 07 Jun 2023 09:05:31 GMT

we have multiple HF"S.

do we need to add props file for this in all of those HF ?

and also how we can add is it by creating new app in the HF

Re: Logs delay due to timezone difference in the UF server

isoutamo — Wed, 07 Jun 2023 09:08:15 GMT

You must add props.conf & transforms.conf into first full Splunk Enterprise instance on path from source system to indexers. If you have several HF on same level/layer where UF are connecting then add those to all of them. Also remember restart splunk on those instances.

Re: Logs delay due to timezone difference in the UF server

sekhar463 — Wed, 07 Jun 2023 09:17:25 GMT

what was the configuration based on my sourcetype = netapp:cloudsecure:agentlog

i have already added props file as mentioned in eariler is anything need to add

transform config file also required here.

Re: Logs delay due to timezone difference in the UF server

sekhar463 — Thu, 19 Oct 2023 12:49:42 GMT

hai,

i have added props.conf file in Heavy forwarders but getting duplicate events for one log which is as CST and UTC check below
please let me know anything need to add.

Re: Logs delay due to timezone difference in the UF server

PickleRick — Sun, 11 Jun 2023 19:27:59 GMT

These are two separate events with two distinct timestamps (other than that they do indeed seem identical).

Since they are 5 hours apart, they probably got ingested via two separate "routes". We don't know your environment and don't know your event routing.