We are collecting logs from Infoblox and forwarding from the product into Splunk which is working as expected, however the timezone Splunk is indexing appears to be in GMT/UTC when the timestamps are actually in EST (when I run a search _time is 4 hours behind). I've gone through the documentation which references setting TZ in props.conf, but this has been unsuccessful so far. Also, Infoblox sends this data over a s2s tcp connection since Splunk is built in which acts as a Universal Forwarder.
Is it possible to set a TZ setting on the Infoblox side before sending logs over the Splunk or am I just missing something in my current configuration to get this to work?
For context, the infoblox DNS events do not have a timezone in the raw event and we are collecting the events in this fashion:
Infoblox (UF) -> HF -> IN
This is also my props.conf settings which are on the HF and IN:
[source::/infoblox/logs*] TZ = US/Eastern [host::servername*] TZ = US/Eastern
This is probably too simple of a suggestion but have you checked what timezone is set for the user in your Search Head's Splunk Web? This will affect the time displayed in your search results.
Check under username > preferences
While changing the user preferences settings does adjust the _time value to the correct value, I still need to search back 4-5 hours just to see real-time events. I think this is an issue with the index time timezone configuration, however since Infoblox has a built-in UF, their appliance controls alot of the props.conf pieces as far as I've been able to tell, but they are limited in their settings. For example, my customer needs to set the index name and sourcetype before sending it over s2s which is a little bit different of a configuration then I've ever worked with.
So just to clarify, you updated props.conf, restarted your instances, and any new incoming data after the change is still showing the wrong time zone?
- I don't see why this would make a difference but I saw a couple of people commenting that it worked for them for the source stanza and not for the host stanza, as well as removing the spaces from the config i.e. TZ=US/Eastern.
Put below props.conf file with the proper values to it on your HF.
[source::/infoblox/logs*] TZ = TIME_PREFIX = TIME_FORMAT = MAX_TIMESTAMP_LOOKAHEAD =
Make sure your machine timings on all search heads, indexers and heavy forwarder is correct. Also, make sure your user preference timings on search heads are okay.
I think you may have misread. I have a timestamp in my events, but nothing that would represent the timezone (ex. UTC-3). Just to test, I also used the above setting prior and saw no change in the event times. All of these settings seem to be controlled through Infoblox.
I've updated the answer. Please try this props.conf and do not forget to re-verify machine timings on all instances.
One more question why you are specifying TZ on both stanzas with host and source? Anyway on one event only one will apply.
Thanks again for your suggestions. I've tried the above settings and adjusted them to what I believe would be the proper settings, but I did not see any changes in my results. The issue appears that when Infoblox's Splunk forwarder (some form of light weight forwarder not a normal UF) sends the data to my HF over s2s, I have no ability to manipulate the data in anyway. Attempting to change anything with this data in the props.conf file on my HF/IN's has had no changes at all.
The reason I have a host stanza is just for demonstration purposes of what I have tried that has no worked. I am trying to show that no settings in the props.conf file for this data source will manipulate the data in any form.
Because this is over s2s and my HF is acting as an Intermediate forwarder, is the data jumping to a queue that neither my IN or HF are able to further parse possibly? I've even tried adjusting the s2s routes to dump this source to the parsingQueue for futher data manipulation, but this has also not worked.
Also, date_zone is showing as "0" on these events and if the events were pulling the timezone from the OS I would expect to see a "local" value in this field so I am ruling that portion out.
Please share your sample event, props.conf and screenshot showing indexer, forwarder machine's timezone, current user timezone preference and sample search result.