To preface my question, I've gone over docs and multiple other questions trying to find a definitive solution, but am still running into a wall. I read through the props.conf documentation, the timezone documentation, and multiple other posts. The answer may be in front of me, but if so I'm missing it and I apologize in advance.
My issue: I have a bunch of devices generating syslog events that are being sent straight to Splunk with no in-between. Cisco switches and routers, Palo Alto firewalls, NTP servers, environmental sensors, and RHEL hosts. All using index:syslog and sourcetype:syslog. While I recognize this is far from ideal, it is the environment I was handed when made the Splunk admin, and I'm trying to work through it. For the most part this works; with enough field-value pair tags, field extractions, and detailed search filters I'm getting the info I need from the hosts. The problem is that a few (12) of our hosts are using GMT as their timezone, while everything else is using the local time (CST) - this is something that cannot be changed, they must use GMT time. Also, the timezone is not identified within the text of the event. It's just a timestamp. Because of this, we're getting events from those hosts that, to Splunk, are occurring six hours in the future, findable only by using (earliest=+1h latest=+7h) in our searches. This isn't viable when trying to look at events from multiple hosts in conjunction.
My fix was to try and add a timezone designation within props.conf, using a regex to identify the hosts affected in a single stanza. I put the regex together and verified it works by running a search using it, which pulled only the hosts I wanted. So, in Splunk/etc/system/local/props.conf I added the stanza:
[host::(doma0wkst*|domsrv(10|11)|192.168.10(12|14|16|18))]
TZ = UTC
To identify the effected hosts (all hosts that started with "doma0wkst", domsrc10 & 11, and 192.168.10.12, .14, .16, .18) and tell Splunk they were reporting UTC time. My understanding was that Splunk would take this and automatically convert the event times to local so that they would align with all the other events we receive. But, this is not working. After adding that and restarting the Splunk service, I'm still getting events from the future. My second thought was to add multiple stanzas, one per host; if that is the best solution, that is what I will do. But I figured I would ask in here to see if there were a better solution first.
