Iguinn, thanks for the response. Will try and address your great questions
Fully understand (i believe) the timestamp process. We do not currently have any issue with anything Splunk, unless I have missed something. I may be looking for a silver bullet, where none exists.
Regarding incoming raw events, I am using and focusing on "table=_internal" as the benchmark for each reporting UF, since I fully believe this to be the most accurate representation of the UF hosting servers clock. That being said, yes, the "date_zone" is being reported as +0700. No abbreviations appear anywhere, which is expected.
To the question of: Is a TZ set for the host? We do not have direct access to any of the current 300 hosts, so TZ settings are out of our control unless within Splunk. None currently appear anywhere.
Yes, as stated in #2, I am using _internal, and everything is v. 6.4.4.
I understand that we can fix the TZ setting using props.conf, but isn't it true that this will have no effect on the original RAW event logs generated by applications external to Splunk, but indexed by Splunk? Two issues I am trying to overcome/address. Here were are concerned with the forensics aspect of the event, i.e. the time within the raw event must agree with the host. Where no time exists within the raw event, then there is no issue
1. First: Validate that the TZ or date_zone agrees with the physical location. Will likely end up writing a query that compares the _internal events with a lookup table.
2. Second: Identify when the "original" event time (Splunk cannot modify this, correct?) is incorrect (same as #1), so that we can have the server owner set their TZ/offset correctly.
Regarding your stated preference to use UTC, I fully concur, but we do not have control of the servers, only Splunk (and even then only via Apps). No login capability.
Finally, to your last question: No problem with this understand, that Splunk displays time per my login setting.
Anybody else out there have to deal with this?
... View more