Splunk Search
Highlighted

Distributed search with multiple timezones

Builder

We recently set up a distributed search environment to display live data on a shared dashboard. The search peer is 3.5hrs ahead of the search head. Is it possible to configure Splunk to use a separate timezone for this particular Splunk server?

0 Karma
Highlighted

Re: Distributed search with multiple timezones

Champion

If you're asking for the incoming data? Yes. Set the TZ on the inputs.conf definition.

If you're asking for display, that's governed by a user's time zone setting (unless the user is authenticated via LDAP). A user can set whatever time zone and see events in their zone. The _time field of an event (what Splunk parses as the time) is in epoch time, so it can be localized to whatever time zone. The log events themselves will still show in their native time zone.

0 Karma
Highlighted

Re: Distributed search with multiple timezones

Builder

Both Splunks (search head and peer) display the desired timezone correctly for reporting purposes. Now I want to display reports on a live dashboard from the peer while keeping the timecharts displayed the same way as I would if I were searching on that Splunk instance.

Currently, my solution is to adjust my searches on the search head as earliest=-0d@d-210m so that I will capture all data I need for the current day and then adjust as necessary.

I was hoping there was a way that I could specify on the search head to adjust all times where splunkserver = SplunkB by 3.5 hours.

0 Karma
Highlighted

Re: Distributed search with multiple timezones

Champion

I'm still not quite following what problem you're trying to solve. You want to report over calendar day?

The thing about epoch time is that epoch time is epoch time is epoch time. That is, 0 in epoch time is midnight Jan 1 1970 in UTC, 4PM on Dec 31 in Pacific time, and 5:30am on Jan 1 in India.

So if you're displaying the calendar day in your time zone, the events from server B will be timeshifted to match your time zone. Why do you feel you need time math to adjust the display? The "live dashboard" part suggests that you want to see "now", so time math would really be contra-indicated.

0 Karma
Highlighted

Re: Distributed search with multiple timezones

Builder

I don't want to display the time in my time zone. I want to display them in the timezone of how each Splunk instance is configured. Then on the a centralized dashboard I will have current day reports for each geographical region (ie some reports based on London time, some with Italy, India, etc)

0 Karma
Highlighted

Re: Distributed search with multiple timezones

Champion

I see.

If the events are localized to a particular server, then you could do second math keyed off of the splunk_server field. Consider: eval _time=if(splunk_server="india", _time - (210 * 60), _time). (_time is in seconds.)

The wrinkle here is that the time range picker would pick events based on _time, and then you'd shift that window, so you'd get blank spots, unless you also adjusted the time range you're searching over.

Highlighted

Re: Distributed search with multiple timezones

Builder

Thanks. Yeah, the issue I'm having is with adjusting the time range on real-time searches for the current day. For example, I want to count the total sessions and unique users by region. I was hoping to have a single search that would capture all events needed and then filter them with something like | eval splunk_server = if(splunk_server="splunk_a" OR (splunk_server="splunk_b" AND _time >= relative_time(now(),"-0d@d")), splunk_server, null()) but it seems this will not work with a real-time search. My best option may be to create multiple real-time reports based on region.

0 Karma
Highlighted

Re: Distributed search with multiple timezones

Champion

I think the per-region report is probably going to be the safest approach. You've got an excellent case for an enhancement request, though.

Consider the classical "newsroom" clocks with one shown for each of several time zones simultaneously. It sounds like your request is basically "allow me to run search 'x' in time zone 'z'".

0 Karma
Highlighted

Re: Distributed search with multiple timezones

Builder

Yes, the newsroom clock is a perfect example.

My only concern with the per-region report is having several real-time reports running. I guess I'll just have to test and monitor system utilization.

0 Karma
Highlighted

Re: Distributed search with multiple timezones

Builder

In the end I was able to get this to work as needed. I used time() instead of now(). It seems that using relative_time(now(),"-0d@d") would not work on a real-time dashboard. I am now able to set my initial time window large enough and then eval the time as necessary.

if(splunk_server="splunk_a" OR (splunk_server="splunk_b" AND _time >= relative_time(time(),"-0d@d")), splunk_server, null())
0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.