All of my events are coming one hour in the future. I've checked the f5 web console and it shows Date:
Apr 26, 2016
1:59 PM (CDT).
I've tried adding the following to the local/props.conf on the heavy forwarder.
[source::lf-app*] TZ = America/Chicago [source::lf-gapp*] TZ = America/Chicago
I've also tried under individual props stanzas.
[f5_bigip:icontrol:locallb] TZ = America/Chicago rename = f5:bigip:ltm:locallb:icontrol
However the events are stilling coming in with an Eastern time.
OK, you definitely HAD a TZ problem but if you have addressed it, you need to wait 3600 seconds (1 hour) before you reassess because your events were being thrown 1 hour into the future. This means that you have to wait for them to trickle into the present (and then to the past) before you will be able to effectively ignore them. You should be doing your evaluation search for a very short window (like
Last 5 minutes). You need to be aware that these changes require that you restart Splunk on your HFs and they only apply to events that arrive at the Indexers/HFs post-restart (what is already done, will stay wrong).
Thanks for your help @woodcock. You are correct. I am still seeing the issue with events approximately one hour into the future. I have restarted after making the props.conf changes.
index=*** sourcetype=*f5** source=*** host="***" earliest=+55m@m latest=+65min@m | head 1000 | dedup 10 index sourcetype source | eval index_time=strftime(_indextime,"%Y-%m-%d %H:%M:%S") | eval index_lag=_indextime-_time | eval event_length=len(_raw) | table index sourcetype source _time index_time index_lag linecount event_length _raw
Just to clarify, my config changes did not correct the issue. Wondering if the rename in the stanza has something to do with it. I may try putting the rules under the renamed sourcetype stanza to see if that makes a difference.