Hi everyone. I have logs that are sent to me in Central Standard Time (-6 hours) but there isn't anything in the TA noting that, so all my logs look like they are 6 hours behind.
As such, I went in and added a props.conf in local
with the statement
[infoblox:dhcp]
TZ = CST
Pushed the updated TA through the cluster bundle on my cluster master to all indexers and verified they all received the updated TA. Looking at my latest logs (about 30 afterwards) I still see the latest logs showing up as 6 hours behind (no change). I ran btool to see which props settings were being picked up by the app and indeed it shows it there.
/opt/splunk/bin/splunk cmd btool --app=Splunk_TA_infoblox props list
[infoblox:dhcp]
EVAL-...
EXTRACT-...
TZ = CST
Any ideas? I feel like I'm overlooking something obvious.
"The sourcetype is named via a transforms, and then I am calling that name in props for the TZ fix."
Well, there's your issue. Splunk doesn't reiterate over props.conf after changing the sourcetype with a transforms. So you cannot do index time stuff like TZ using the sourcetype you set by transforms. You need to do it on the original sourcetype (or source or host).
@DEAD_BEEF Here is a way to re-iterate over the props.conf, basically addressing the same issue you've raised:
Like this:
[YourOriginalSourcetypeHere]
# DO NOT USE [infoblox:dhcp]
TZ = CST
Or like this:
[source::/Your/Path/And/Source/Here]
TZ = CST
Or like this:
[host::YourHostHere]
TZ = CST
"The sourcetype is named via a transforms, and then I am calling that name in props for the TZ fix."
Well, there's your issue. Splunk doesn't reiterate over props.conf after changing the sourcetype with a transforms. So you cannot do index time stuff like TZ using the sourcetype you set by transforms. You need to do it on the original sourcetype (or source or host).
True. You nailed it.
So just to be clear. The indexer is performing a regex, when there's a match, it puts those "hits" into a sourcetype called "foo". I was then calling foo in my props.conf to change the timezone. What @FrankVl is saying is that after performing the transform, splunk won't then use my TZ setting in props?
Is this an order of operations thing for the .conf files? I can't do it on the original sourcetype as there is no former sourcetype. I'll have to look at the host or source, but I'm worried that there are two sourcetypes that get regex'd from the data so source and/or host may not work either. I'll look into this more to see if these are feasible.
Data enters the parsing stage with a certain sourcetype/source/host based on the input stage. That is used to determine what props and transforms to apply. Then all those are applied in a certain order.
If any of those props or transforms change sourcetype/source/host values, splunk does not go back to the start of the parsing phase to again determine what props and transforms to apply.
It may very well be that you will have to look at splitting the data before it reaches Splunk, such that Splunk can properly apply the relevant config to each different flavor.
@FrankVl oh ok, that makes sense. Since the sourcetype is changed by the transforms, it must have already read the TZ props (beforehand) and the foo sourcetype doesn't exist yet so nothing to apply it to. Thanks for the clarification.
How are the logs coming in to your indexers? Is there a heavy forwarder involved by any chance?
Logs are being sent to a syslog server, UF running on that sending it to the indexers. Splunk TA is then on the indexer doing the magic.
The precedence for TZ is:
1: If %z
is configured in TIME_FORMAT
and is in the event use that EXCEPT...
0: If this matches a TZ_ALIAS
setting, override with that.
2: If the forwarder has a TZ
setting, use that.
3: If the indexer has a TZ
setting, use that.
4: Use the TZ that is configured in the host OS of the indexer (which could be different on every indexer).
You can get some hits as to what has happened for your event by checking the date_zone
field.
Much of the time the actual problem is that you are not testing your configuration changes properly so BE ABSOLUTELY SURE that:
1: You restart the splunk service on any device where you deploy a configuration.
2: Add _index_earliest=-1m
to your search to make absolutely certain that you are looking ONLY at events that were recently indexed.
Hi @woodcock!
1: No TIME_FORMAT
0: No TZ_ALIAS
If I use the TZ that is configured in the indexer (3) or OS (4), wouldn't that set TZ = GMT since both my indexer and the OS is running in GMT? These logs are coming in as CST. I'm sorry I don't understand these suggestions.
I didn't restart the splunk service because when I did --check-restart
it said restart is not required since I am only adding a new props.conf
.
Tried using _index_earliest=-1m
and no results. Neat command!
The TZ is set at indextime and is immutable after that. Your TZ
configuration changes will never effect already-indexed data (which is why I mentioned that your testing assumptions are probably where the problem is). You must send NEW data in to see if your new settings work.
I'm getting new data in constantly. Are you saying that the new data won't have the new props TZ applied to it for some reason?
I am saying if something like _index_earliest=-1m
and a timepicker of All time
returns no events, then you are not getting new events all the time. Did you use All time
?
@woodcock okay, sorry for my confusion. I ran _index_earliest=-1m
for all time and it is showing new events that are "6 hours behind". I refreshed it a few times and verified that the event count is increasing and getting newer timestamped events.
Run this command on your indexers:
splunk btool props list --debug | less
It should help pinpoint what value is being used AND FROM WHAT FILE.
from running the command I found this in the output:
/$splunk_home$/Splunk_TA_infoblox/local/props.conf TZ = CST
That is the only output for TZ
? That is way unexpected.
Are you using INDEXED_EXTRACTIONS
or Heavy Forwarders on this input?
If the former, this setting needs to be on the UF, if the latter, this setting needs to be on the HFs.
It was the only TZ for that app. there were other hits for TZ
but they were all in other apps on the indexer (none in /etc/system/default or /etc/system/local).
This is an indexer with the Splunk TA running. The sourcetype is named via a transforms, and then I am calling that name in props for the TZ fix. No INDEXED_EXTRACTIONS
, the data is coming in via .log files from a UF.
It doesn't matter what app it is in, it matters what sourcetype it is applied to. In any case, try sending the props.conf to the UF and BE SURE that the sourcetype
value for this setting (regardless of what app contains it) matches your data. I suspect that this is your problem.