How can I fix logs delay due to timezone difference in the UF server?

[sekhar463]

Hi All i have a log source in the server timezone is in CST and logs are coming into the server as UTC time zone logs.

so while ingesting logs splunk was ingesting based on CST which is 5 hours before logs.

i have added props config for UTC but still getting the issue.

please let us know what needs to be done

[PickleRick]

There can be many things that went wrong and you're not providing too much information 😉

@isoutamowas a tiny bit imprecise saying that splunk uses UTC internally. Actually splunk internally stores _time as a "unix timestamp" which just points to a given point in time and can be rendered to a human-readable time string in any timezone. In order to do so Splunk has to parse the human readable (or not) timestamp from the event.

So the problem with event's time can be in three separate places (in each of them or even if all three of them at the same time).

1. The source logs can have wrong timestamp (happens with badly configured source - for example, if someone configured a device to store local time but report it as UTC)

2. The log is wrongly parsed due to misconfiguration (or if there was no explicit configuration at all and Splunk's "guess" as to the timestamp format turns out to be wrong).

2a. There is no timezone information present in the event and splunk applies wrong TZ to the timesamp.

3. The user's timezone is misconfigured in preferences and the timestamp is wrongly rendered in the webui.

So depending on what is the problem, there should be proper fix

1. Configure source properly

2. Apply time extraction configuration on the indexers (or HFs if you have them) - in a typical example

2a. Add TZ definition for a given sourcetype/source/host - this can be a bit more tricky than what @diogofgm showed if you have multiple event sources from one host - each with different TZ

3. Change your user's preferences.

[diogofgm]

You can create a stanza in props.conf with:

[host::<host>]
TZ=<TZ>

[sekhar463]

i have already added props file with sourcetype but didnt fixed.

[netapp:cloudsecure:agentlog]
TZ = UTC

[PickleRick]

But on which component did you put that setting?

[sekhar463]

i have added props.conf in the Deployment server for the specific app to update in the UF 

[PickleRick]

And that's why it's not working 🙂

UF is not a parsing component. UF just picks up data, splits it up into chunks and sends to indexer(s) or intermediate forwarder(s). You need that setting on your parsing components (indexers or heavy forwarders) in the path of your events.

[sekhar463]

hai,

i have added props.conf file in Heavy forwarders but getting duplicate events for one log which is as CST and UTC check below 
please let me know anything need to add.


2023-06-11T15:45:31,331 [INFO] [prod] [2f78061f-5f51-4636-8da1-3c9644b9e7a1] [87effa85-dec0-4084-af7b-49404cd5d1d4] [agent-AgentDataSourceStateManagerActor] - All collector health status has been updated- stateMap: [Map(1b90fa7b-60a6-4cab-8c58-a3a17751f0e4 -> running, c2e84623-8360-4c92-a238-0e505bce55d2 -> running, c3b0cd61-9ed1-4e91-a356-a013a148ae7d -> running, ec90ff7c-9414-4791-b42a-671023538299 -> error, 0f02f369-1fd0-4184-845f-9b9a30b47ef8 -> running, 413a8ff6-3504-4321-a8a7-2f3f88c0e9d5 -> running, 4e500dc7-75c3-4d92-b04c-1b717a094bd2 -> running)], statusMap: [Map(1b90fa7b-60a6-4cab-8c58-a3a17751f0e4 -> Collector is running., c2e84623-8360-4c92-a238-0e505bce55d2 -> Collector is running., c3b0cd61-9ed1-4e91-a356-a013a148ae7d -> Collector is running., ec90ff7c-9414-4791-b42a-671023538299 -> Connector is in error state. Service name: audit. Reason for failure: External fpolicy server terminated., 0f02f369-1fd0-4184-845f-9b9a30b47ef8 -> Collector is running., 413a8ff6-3504-4321-a8a7-2f3f88c0e9d5 -> Collector is running., 4e500dc7-75c3-4d92-b04c-1b717a094bd2 -> Collector is running.)]
host = usinfnacild100.ops.invesco.netsource = test sourcetype = test1
6/11/23
9:15:31.322 PM
2023-06-11T10:45:31,322 [INFO] [prod] [2f78061f-5f51-4636-8da1-3c9644b9e7a1] [87effa85-dec0-4084-af7b-49404cd5d1d4] [agent-AgentDataSourceStateManagerActor] - All collector health status has been updated- stateMap: [Map(1b90fa7b-60a6-4cab-8c58-a3a17751f0e4 -> running, c2e84623-8360-4c92-a238-0e505bce55d2 -> running, c3b0cd61-9ed1-4e91-a356-a013a148ae7d -> running, ec90ff7c-9414-4791-b42a-671023538299 -> error, 0f02f369-1fd0-4184-845f-9b9a30b47ef8 -> running, 413a8ff6-3504-4321-a8a7-2f3f88c0e9d5 -> running, 4e500dc7-75c3-4d92-b04c-1b717a094bd2 -> running)], statusMap: [Map(1b90fa7b-60a6-4cab-8c58-a3a17751f0e4 -> Collector is running., c2e84623-8360-4c92-a238-0e505bce55d2 -> Collector is running., c3b0cd61-9ed1-4e91-a356-a013a148ae7d -> Collector is running., ec90ff7c-9414-4791-b42a-671023538299 -> Connector is in error state. Service name: audit. Reason for failure: External fpolicy server terminated., 0f02f369-1fd0-4184-845f-9b9a30b47ef8 -> Collector is running., 413a8ff6-3504-4321-a8a7-2f3f88c0e9d5 -> Collector is running., 4e500dc7-75c3-4d92-b04c-1b717a094bd2 -> Collector is running.)]
host = usinfnacild100.ops.invesco.net source = test sourcetype = test1

[PickleRick]

These are two separate events with two distinct timestamps (other than that they do indeed seem identical).

Since they are 5 hours apart, they probably got ingested via two separate "routes". We don't know your environment and don't know your event routing.

[sekhar463]

we have multiple HF"S.

do we need to add props file for this in all of those HF ?

and also how we can add is it by creating new app in the HF 

[isoutamo]

You must add props.conf & transforms.conf into first full Splunk Enterprise instance on path from source system to indexers. If you have several HF on same level/layer where UF are connecting then add those to all of them. Also remember restart splunk on those instances.

[sekhar463]

what was the configuration based on my  sourcetype = netapp:cloudsecure:agentlog 

i have already added props file as mentioned in eariler is anything need to add 

transform config file also required here.

[isoutamo]

Hi

Splunk use internally UTC for all times. When you are ingesting event splunk automatically change it's time to UTC based on your source system timezone and if it cannot determine it when events are collected then it use indexers time zone. When you are searching then it's use your web browser's time zone unless you haven't separately defined it in your Splunk's User preferences.

If your serves (UF etc) haven't correct TZ information and time is not in sync, you should fix this. Another option is use conf files on UF and Indexer/HF side to fix this issue.

More about timezone handling in splunk https://docs.splunk.com/Documentation/Splunk/latest/Search/Abouttimezones

r. Ismo