But the use case is a bit larger than this. Let's say I have people who need to run searches that live on 3 continents but they all need global visibility into the data that lives in indexers in 3 different global data centers. Their job is to make sure that all locations are fufulling time based (relative to the local time zone of the data center) SLAs and would like to have a single view that shows all of these events.
A 'simple' search example would be show me all transactions that finished later than 9:15AM wherever they may occurred.
The 'best' answer is to log into 3 different search heads in each location and get the results, but what if you have 10 DCs and/or timezones? The 'hack' is to mess with timezone settings at the indexer lever and use one search head, which has it own issues of breaking time series correlation.
It's painfully close and seems like Splunk should have a search command, function or subset of distributed search that could ignore the offsets imposed during a default distributed search and take the actual hour of the event. Any ideas?
I'm not sure if this is correct on my assumption, but, Splunk can search based on epoch time, and since epoch is a reference to GMT, in theory it would be the exact same time no matter what the server is that indexed the data. Have you tried just using the epoch time for your "earliest" and "latest" dashboard search?
index="main" "*Blah*" earliest=1292544000
would search for Blah at midnight GMT on dec. 17. Not sure it will accomplish what you're looking for, but might work.