We have multiple Cisco WSA devices set up in each of the US timezones; each is set to log in local time. But it seems as if the WSA logs don't contain any kind of timezone indicator on them.
When I run a search in Splunk, using a user account in Central time, against a WSA device in Eastern time, I end up getting "future" events.
Example: ran a search at 8AM central against an eastern WSA device, there were events found with time-stamps of 9AM.
Likewise when I run a search looking for lag between index time and timestamp (again from a Central Time account):
index=wsa_system sourcetype="cisco:wsa:shd" CliConn=* | eval lag=((_indextime-_time)/(60*60))
All our Eastern devices are reporting negative lag (future timestamps), Central devices are relatively real-time, Mountain devices have approx 1hr lag, Western devices have roughly 2hr lag.
I'm trying to set up alerts for high numbers of client connections and need to know:
1) Is there any way to adjust for these time off-sets at search time using our current logs?
2) Is there a way for Splunk to add the time off-sets/zone to the events at indexing time?
3) Is there a way to have the WSA devices add the timezone to the logs before sending? (Or will I need to make a business case that all the WSA devices should log in the same timezone regardless of physical location?)
