Deployment Architecture

Is it possible to have different search head cluster member timezones?

Motivator

Is it possible to have a search head cluster in which some of the members have a different time zone?

Documentation only mentions accurate time ie via ntp but not actual local time zones.
Say I have 5 machines all in the same data center but we want the default time on 2 to be different.

ie, I have 3 search heads in timezone (gmt +1 hour) and 2 search heads in another (adhoc search only gmt +2 hours).
By adding adhoc search only to the ones differing we stop jobs from running at the wrong time.

Could this function properly without issue. ie. no issues with captain synchronization etc? job/artifact timeouts etc.

0 Karma

Splunk Employee
Splunk Employee

Splunk the software may currently permit this configuration, but it should not be used. Among other possibilities, scheduled search time-to-run expressions are not timezone-independent.

Splunk Employee
Splunk Employee

Timezones should be set the same on each SHC member, this time we dont have site awareness for SHC.

For time zone adjustments, you could have your users adjust the their home time zone. However, if your SHC config is behind a load balancer and unless you're using src-ip forwarding, you cannot control which search head your users get. So the time zone difference in settings isnt management per search head.

0 Karma

Splunk Employee
Splunk Employee

Is the claim here that for usability reasons you should not set up members with different timezones? It seems like you could definitely have a case where particular users are directed to particular sets of search heads, and you could also configure all users with per-user timezones.

So I don't understand why this is needed.

Splunk Employee
Splunk Employee

This is a followup on multiple timezones in SHC. It sounds like Lucas has his issue resolved, but the posting brought up some other questions that were deserving of clarification.

Search heads in SHC ~need~ to all be on the same timezone to make things work right, UTC is always nice, but as long as they are in agreement it should be fine.

Here is a scenario Search/report scheduled 2AM ( 0 2 * * *), user is on a SHC where captain is in PST, so it will run at 2AM PST. If the captain fails at 1AM PST and goes to a captain in EST. The search will not run (since the time is already past as far as the captain is concerned). If it fails the other way EST->PST at 2:30am, will get it running twice. This is due to the fact that cron jobs are not timezone aware.

As it was mentioned, if there is need to view data with different timezones, it should be done at the user profile level.

Legend

I agree @jrodman - I would configure the users with timezones. And usually each user can change/set their own timezone as desired at any time. Further, if a user ended up on a different search head tomorrow, he/she would still see the user-selected timezone.

Having different OS timezones set for the different SHC members seems like it might have unexpected consequences - both for execution of background tasks and for any users running with "system default timezone" selected.

0 Karma

Splunk Employee
Splunk Employee

Okay following up to myself:
One reason that a cluster should have agreeing timzeones is that search scheduling is distributed across the cluster. However, the time to execute searches is expressed in the local timezone of the search head. This means you can have problems like a user scheduling a search at one time, but the scheduler evaluating it in a different timezone. Also, when the captaincy of the cluster moves, it could change timezones, which I cannot even imagine a strategy to handle.

Motivator

We have a special circumstance where we've HAD to do a double install.

An indexer and a search head cluster member on the same machines.

Indexers are on UTC time by default. This leads to an issue with the search head instance being in the UTC timezone by default.

In trying to remedy this I was planning on rolling in more cluster members (in another timezone) and rolling out the ones in UTC.

0 Karma

Motivator

We've figured out how we can do this. Change incoming shc members to utc. Wait for everything to sync. Roll out old members.
Stop all members and change them to our timezone and start them up again.

0 Karma

Splunk Employee
Splunk Employee

This is a bit odd. I'm not aware of any requirement that search heads have timezones that agree with indexers. I'd prefer it for administration sanity however.

0 Karma