If you're asking for the incoming data? Yes. Set the TZ on the inputs.conf definition.
If you're asking for display, that's governed by a user's time zone setting (unless the user is authenticated via LDAP). A user can set whatever time zone and see events in their zone. The _time field of an event (what Splunk parses as the time) is in epoch time, so it can be localized to whatever time zone. The log events themselves will still show in their native time zone.
Both Splunks (search head and peer) display the desired timezone correctly for reporting purposes. Now I want to display reports on a live dashboard from the peer while keeping the timecharts displayed the same way as I would if I were searching on that Splunk instance.
Currently, my solution is to adjust my searches on the search head as
earliest=-0d@d-210m so that I will capture all data I need for the current day and then adjust as necessary.
I was hoping there was a way that I could specify on the search head to adjust all times where splunkserver = SplunkB by 3.5 hours.
I'm still not quite following what problem you're trying to solve. You want to report over calendar day?
The thing about epoch time is that epoch time is epoch time is epoch time. That is, 0 in epoch time is midnight Jan 1 1970 in UTC, 4PM on Dec 31 in Pacific time, and 5:30am on Jan 1 in India.
So if you're displaying the calendar day in your time zone, the events from server B will be timeshifted to match your time zone. Why do you feel you need time math to adjust the display? The "live dashboard" part suggests that you want to see "now", so time math would really be contra-indicated.
I don't want to display the time in my time zone. I want to display them in the timezone of how each Splunk instance is configured. Then on the a centralized dashboard I will have current day reports for each geographical region (ie some reports based on London time, some with Italy, India, etc)
If the events are localized to a particular server, then you could do second math keyed off of the splunk_server field. Consider:
eval _time=if(splunk_server="india", _time - (210 * 60), _time). (_time is in seconds.)
The wrinkle here is that the time range picker would pick events based on _time, and then you'd shift that window, so you'd get blank spots, unless you also adjusted the time range you're searching over.
Thanks. Yeah, the issue I'm having is with adjusting the time range on real-time searches for the current day. For example, I want to count the total sessions and unique users by region. I was hoping to have a single search that would capture all events needed and then filter them with something like
| eval splunk_server = if(splunk_server="splunk_a" OR (splunk_server="splunk_b" AND _time >= relative_time(now(),"-0d@d")), splunk_server, null()) but it seems this will not work with a real-time search. My best option may be to create multiple real-time reports based on region.
I think the per-region report is probably going to be the safest approach. You've got an excellent case for an enhancement request, though.
Consider the classical "newsroom" clocks with one shown for each of several time zones simultaneously. It sounds like your request is basically "allow me to run search 'x' in time zone 'z'".
Yes, the newsroom clock is a perfect example.
My only concern with the per-region report is having several real-time reports running. I guess I'll just have to test and monitor system utilization.
In the end I was able to get this to work as needed. I used
time() instead of
now(). It seems that using
relative_time(now(),"-0d@d") would not work on a real-time dashboard. I am now able to set my initial time window large enough and then eval the time as necessary.
if(splunk_server="splunk_a" OR (splunk_server="splunk_b" AND _time >= relative_time(time(),"-0d@d")), splunk_server, null())