Getting Data In

Host timezone offset, fix in props.conf

Gamer0364
Loves-to-Learn

To preface my question, I've gone over docs and multiple other questions trying to find a definitive solution, but am still running into a wall. I read through the props.conf documentation, the timezone documentation, and multiple other posts. The answer may be in front of me, but if so I'm missing it and I apologize in advance.

My issue: I have a bunch of devices generating syslog events that are being sent straight to Splunk with no in-between. Cisco switches and routers, Palo Alto firewalls, NTP servers, environmental sensors, and RHEL hosts. All using index:syslog and sourcetype:syslog. While I recognize this is far from ideal, it is the environment I was handed when made the Splunk admin, and I'm trying to work through it. For the most part this works; with enough field-value pair tags, field extractions, and detailed search filters I'm getting the info I need from the hosts. The problem is that a few (12) of our hosts are using GMT as their timezone, while everything else is using the local time (CST) - this is something that cannot be changed, they must use GMT time. Also, the timezone is not identified within the text of the event. It's just a timestamp. Because of this, we're getting events from those hosts that, to Splunk, are occurring six hours in the future, findable only by using (earliest=+1h latest=+7h) in our searches. This isn't viable when trying to look at events from multiple hosts in conjunction.

My fix was to try and add a timezone designation within props.conf, using a regex to identify the hosts affected in a single stanza. I put the regex together and verified it works by running a search using it, which pulled only the hosts I wanted. So, in Splunk/etc/system/local/props.conf I added the stanza:

[host::(doma0wkst*|domsrv(10|11)|192.168.10(12|14|16|18))]
TZ = UTC

To identify the effected hosts (all hosts that started with "doma0wkst", domsrc10 & 11, and 192.168.10.12, .14, .16, .18) and tell Splunk they were reporting UTC time. My understanding was that Splunk would take this and automatically convert the event times to local so that they would align with all the other events we receive. But, this is not working. After adding that and restarting the Splunk service, I'm still getting events from the future. My second thought was to add multiple stanzas, one per host; if that is the best solution, that is what I will do. But I figured I would ask in here to see if there were a better solution first.

Labels (2)
0 Karma

somesoni2
Revered Legend

Try adding stanza for one of the host pattern and see if that works. 

If it does, try adding another pattern into your regex and see if both still works and keep adding more patterns.

Also, try this one as well

[host::(doma0wkst|domsrv(10|11)|192.168.10.(12|14|16|18))*]
TZ = UTC
0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...