Splunk Standalone Search Head Default TZ Settings

[JGP]

We want to set default TZ as SGT for a particular Search Head and that SH is in EDT TZ. We have already applied TZ setting in props settings at master for that index so they can view the related events when it is pushed.

Now, application team wants in preferences settings it should be SGT as default in preferences settings so whenever any query is search for the index it should show as SGT TZ. as it can be seen in the sample events which is not coming as expected. 

here is the btool results for the SH

-bash-4.2$ /opt/splunk/splunk_sas/bin/splunk btool --debug user-prefs list
/opt/splunk/splunk_sas/etc/system/local/user-prefs.conf [default]
/opt/splunk/splunk_sas/etc/system/local/user-prefs.conf tz = Asia/Hong_Kong
/opt/splunk/splunk_sas/etc/apps/splunk_instrumentation/local/user-prefs.conf [general]
/opt/splunk/splunk_sas/etc/apps/splunk_instrumentation/local/user-prefs.conf dismissedInstrumentationOptInVersion = 4
/opt/splunk/splunk_sas/etc/system/local/user-prefs.conf hideInstrumentationOptInModal = 1
/opt/splunk/splunk_sas/etc/system/local/user-prefs.conf notification_python_3_impact = false
/opt/splunk/splunk_sas/etc/system/local/user-prefs.conf render_version_messages = 0
/opt/splunk/splunk_sas/etc/apps/user-prefs/default/user-prefs.conf search_assistant = compact
/opt/splunk/splunk_sas/etc/apps/user-prefs/default/user-prefs.conf search_auto_format = 0
/opt/splunk/splunk_sas/etc/apps/user-prefs/default/user-prefs.conf search_line_numbers = 0
/opt/splunk/splunk_sas/etc/apps/user-prefs/default/user-prefs.conf search_syntax_highlighting = light
/opt/splunk/splunk_sas/etc/apps/user-prefs/default/user-prefs.conf search_use_advanced_editor = 1
/opt/splunk/splunk_sas/etc/apps/user-prefs/default/user-prefs.conf theme = enterprise
/opt/splunk/splunk_sas/etc/apps/TA_***_LDAP/default/user-prefs.conf tz = GMT
/opt/splunk/splunk_sas/etc/apps/user-prefs/default/user-prefs.conf [general_default]
/opt/splunk/splunk_sas/etc/apps/user-prefs/default/user-prefs.conf appOrder = search
/opt/splunk/splunk_sas/etc/system/local/user-prefs.conf default_earliest_time = -24h@h
/opt/splunk/splunk_sas/etc/apps/user-prefs/default/user-prefs.conf default_latest_time = now
/opt/splunk/splunk_sas/etc/apps/user-prefs/default/user-prefs.conf default_namespace = $default
/opt/splunk/splunk_sas/etc/apps/user-prefs/default/user-prefs.conf hideInstrumentationOptInModal = 0
/opt/splunk/splunk_sas/etc/apps/user-prefs/default/user-prefs.conf notification_noah_upgrade = true
/opt/splunk/splunk_sas/etc/apps/user-prefs/default/user-prefs.conf notification_python_2_removal = false
/opt/splunk/splunk_sas/etc/apps/user-prefs/default/user-prefs.conf notification_python_3_impact = false
/opt/splunk/splunk_sas/etc/apps/user-prefs/default/user-prefs.conf showWhatsNew = 1
/opt/splunk/splunk_sas/etc/system/local/user-prefs.conf tz = Asia/Hong_Kong
/opt/splunk/splunk_sas/etc/apps/TA_***_LDAP/default/user-prefs.conf [role_app_splunk_admin]
/opt/splunk/splunk_sas/etc/apps/TA_***_LDAP/default/user-prefs.conf default_namespace = search
/opt/splunk/splunk_sas/etc/system/local/user-prefs.conf tz = Asia/Hong_Kong
/opt/splunk/splunk_sas/etc/apps/TA_***_LDAP/default/user-prefs.conf [role_app_splunk_api]
/opt/splunk/splunk_sas/etc/apps/TA_***_LDAP/default/user-prefs.conf default_namespace = search
/opt/splunk/splunk_sas/etc/system/local/user-prefs.conf tz = Asia/Hong_Kong
/opt/splunk/splunk_sas/etc/apps/TA_***_LDAP/default/user-prefs.conf [role_app_splunk_***]
/opt/splunk/splunk_sas/etc/apps/TA_***_LDAP/default/user-prefs.conf default_namespace = search
/opt/splunk/splunk_sas/etc/system/local/user-prefs.conf tz = Asia/Hong_Kong
/opt/splunk/splunk_sas/etc/apps/TA_***_LDAP/default/user-prefs.conf [role_app_splunk_infra]
/opt/splunk/splunk_sas/etc/apps/TA_***_LDAP/default/user-prefs.conf default_namespace = search
/opt/splunk/splunk_sas/etc/system/local/user-prefs.conf tz = Asia/Hong_Kong
/opt/splunk/splunk_sas/etc/apps/TA_***_LDAP/default/user-prefs.conf [role_app_splunk_power]
/opt/splunk/splunk_sas/etc/apps/TA_***_LDAP/default/user-prefs.conf default_namespace = search
/opt/splunk/splunk_sas/etc/system/local/user-prefs.conf tz = Asia/Hong_Kong
/opt/splunk/splunk_sas/etc/apps/TA_***_LDAP/default/user-prefs.conf [role_general]
/opt/splunk/splunk_sas/etc/apps/TA_***_LDAP/default/user-prefs.conf default_namespace = search
/opt/splunk/splunk_sas/etc/system/local/user-prefs.conf tz = Asia/Hong_Kong
/opt/splunk/splunk_sas/etc/apps/TA_***_LDAP/default/user-prefs.conf [role_general_default]
/opt/splunk/splunk_sas/etc/apps/TA_***_LDAP/default/user-prefs.conf appOrder = search
/opt/splunk/splunk_sas/etc/apps/TA_***_LDAP/default/user-prefs.conf default_namespace = search
/opt/splunk/splunk_sas/etc/system/local/user-prefs.conf tz = Asia/Hong_Kong

[JGP]

tried adding settings in below file, but didn't worked

tz=Asia/Singapore

/opt/splunk/splunk_sas/etc/apps/user-prefs/local/user-prefs.conf

/opt/splunk/splunk_sas/etc/apps/TA_****_LDAP/default/user-prefs.conf

also tried in below file but no luck(only in default when added but not worked)

/opt/splunk/splunk_sas/etc/apps/TA_****_LDAP/default/ui-prefs.conf

[search]
tz=Asia/Singapore
dispatch.earliest_time = -4h
dispatch.latest_time = now

[default]
tz=Asia/Singapore
dispatch.earliest_time = -4h
dispatch.latest_time = now

[PickleRick]

To be fully honest, I've never done that myself (and I don't see how those settings would get inherited across different roles). I suppose it's one of those things that you're supposed to use UI for.

[JGP]

seems provided link is not accessible. and tried adding in as well, but didn't worked.
/opt/splunk/splunk_sas/etc/apps/user-prefs/local/user-prefs.conf

[default]
TZ=Asia/Singapore

As it is case sensitive, could let me know in which path/filename needs to be added? appreciate your feedback on this

[PickleRick]

Sorry, I missed one slash from the URL. This one should be OK.

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Wheretofindtheconfigurationfiles

As the setting names are case-sensitive, you should set

tz = something

Not

TZ = something

[JGP]

Tried editing one by one in below files but didn't worked. 

[default]
TZ=Asia/Singapore

/opt/splunk/splunk_sas/etc/apps/user-prefs/default/user-prefs.conf

/opt/splunk/splunk_sas/etc/apps/TA_****_LDAP/default/user-prefs.conf

/opt/splunk/splunk_sas/etc/system/local/ui-prefs.conf

[PickleRick]

Apart from possible issues with config files precedence (especially since it's search time so user's context and export settings come into play. See https://docs.splunk.com/Documentation/Splunk/latestAdmin/Wheretofindtheconfigurationfiles ), the option here is called "tz", not "TZ" (the "TZ" option is used in props.conf). Yes, Splunk is case-sensitive when it comes to option names.

[JGP]

Okay. Now consider this case, if there is any no props settings applied and events are flowing to the index with TZ available in the logs with EDT/GMT and also SH is in that Zone too and other splunk components in the same zone  as well, so now if any user wants to view related events in ASIA TZ as if he change the user preference setting to Asia then will show the events in that TZ settings.

So is there any possibility to change the default user preference setting to other which any user wants.  

[PickleRick]

OK. If there are no time-related settings applied to the sourcetype... splunk will start guessing. It might guess the timestamp format and location right (and parse the timestamp properly), it might not guess it right and either parse the timestamp wrongly (for example, leaving out the timezone) or completely miss it and set a timestamp according to the rules outlined here:

https://docs.splunk.com/Documentation/Splunk/latest/Data/HowSplunkextractstimestamps

You might try to manipulate users' settings to some extent with user-prefs.conf but I've never done that. There is no WebUI section for this level of tweaking.

[JGP]

okay, is there any precedence that will be considered here if applied TZ first and then DATETIME_CONFIG next? 

Also, is there is any option to change the default Timezone for user settings in preference as Splunk components will be EDT/GMT TZ.

[PickleRick]

Again - if DATETIME_CONFIG=CURRENT, there is no parsing timestamps so TZ has no meaning. What would you expect it to do if you already got the current timestamp from the operating system? For example, I'm writing it at timestamp 1679596293. It has nothing to do with any timezones.

A default timezone for a newly created user is - if I remember correctly - the timezone set in the operating system for the splunkd process.

[JGP]

thanks for the reply. say for eg , we have applied below settings in props for one of the ST, so in this case it will consider a file as system time and then will convert to TZ settings, am i missing here something?

DATETIME_CONFIG = CURRENT
TZ=Asia/Singapore

[PickleRick]

If you set DATETIME_CONFIG to CURRENT, TZ doesn't matter. The time will be set according to the timestamp of the host it's being ingested on. Whether this timestamp would be rendered as 00:00 UTC, 01:00 CET or 09:00 JST, it's still the same timestamp and it will be displayed in user's interface according to that user's timezone (so if the user has his timezone set as UTC, it will be displayed as 00:00, if he's in JST it will be shown as 09:00 and so on).

[PickleRick]

You're mixing different things here.

It works like this:

1. The event is received on an input in an indexer (or Heavy Forwarder, depending which is first in event's path). The timestamp is getting parsed from the event. If there is no timezone information in the event itself (or the defined time format does not include timezone), the timezone from the input configuration, or the default one for the parsing server is applied. The timestamp is stored in the _time field as unix timestamp (number of seconds since epoch).

2. The event is (if needed) sent further down the pipeline until it reaches the indexer. It's _not_ parsed again. As it reaches the destination indexer it's getting written to an index.

3. Search head searches for an event and retrieves results. The result has _time field (still specified as unix timestamp).

4. SH displays that timestamp according to the timezone defined for the user running the search.

So the timezone settings you apply on indexer (if you don't have any intermediate heavy forwarders) only affect parsing the time from the raw event. They don't change the way the resulting timestamp is displayed in user's browser.

So the question is where did you put that configuration of yours. You're talking about users' settings but say about "master for that index" whatever that would mean.

Another issue I see here is that most probably your timestamp is wrongly parsed from the event (or even not parsed from event at all). If you had a timezone difference, the timestamp contained within the event and the one shown in the webui would differ by full hours with maybe half hour "ending" for some timezones (India?). But in your case one timestamp is "7:25:40", another one is "8:51:50" so the time is completely off.

Before you start fiddling with your users' configured timezones fix your data onboarding.