Getting Data In

Splunk not picking up on timezone set within props

DEAD_BEEF
Builder

Hi everyone. I have logs that are sent to me in Central Standard Time (-6 hours) but there isn't anything in the TA noting that, so all my logs look like they are 6 hours behind.

As such, I went in and added a props.conf in local with the statement

[infoblox:dhcp]
TZ = CST

Pushed the updated TA through the cluster bundle on my cluster master to all indexers and verified they all received the updated TA. Looking at my latest logs (about 30 afterwards) I still see the latest logs showing up as 6 hours behind (no change). I ran btool to see which props settings were being picked up by the app and indeed it shows it there.

/opt/splunk/bin/splunk cmd btool --app=Splunk_TA_infoblox props list

[infoblox:dhcp]
EVAL-...
EXTRACT-...
TZ = CST

Any ideas? I feel like I'm overlooking something obvious.

0 Karma
1 Solution

FrankVl
Ultra Champion

"The sourcetype is named via a transforms, and then I am calling that name in props for the TZ fix."

Well, there's your issue. Splunk doesn't reiterate over props.conf after changing the sourcetype with a transforms. So you cannot do index time stuff like TZ using the sourcetype you set by transforms. You need to do it on the original sourcetype (or source or host).

View solution in original post

efika
Path Finder

@DEAD_BEEF  Here is a way to re-iterate over the props.conf, basically addressing the same issue you've raised:

https://www.linkedin.com/pulse/how-make-splunk-heavy-forwarder-reiterate-over-after-changing-efi/?tr... 

0 Karma

woodcock
Esteemed Legend

Like this:

[YourOriginalSourcetypeHere]
# DO NOT USE [infoblox:dhcp]
TZ = CST

Or like this:

[source::/Your/Path/And/Source/Here]
TZ = CST

Or like this:

[host::YourHostHere]
TZ = CST
0 Karma

FrankVl
Ultra Champion

"The sourcetype is named via a transforms, and then I am calling that name in props for the TZ fix."

Well, there's your issue. Splunk doesn't reiterate over props.conf after changing the sourcetype with a transforms. So you cannot do index time stuff like TZ using the sourcetype you set by transforms. You need to do it on the original sourcetype (or source or host).

View solution in original post

woodcock
Esteemed Legend

True. You nailed it.

0 Karma

DEAD_BEEF
Builder

So just to be clear. The indexer is performing a regex, when there's a match, it puts those "hits" into a sourcetype called "foo". I was then calling foo in my props.conf to change the timezone. What @FrankVl is saying is that after performing the transform, splunk won't then use my TZ setting in props?

Is this an order of operations thing for the .conf files? I can't do it on the original sourcetype as there is no former sourcetype. I'll have to look at the host or source, but I'm worried that there are two sourcetypes that get regex'd from the data so source and/or host may not work either. I'll look into this more to see if these are feasible.

0 Karma

FrankVl
Ultra Champion

Data enters the parsing stage with a certain sourcetype/source/host based on the input stage. That is used to determine what props and transforms to apply. Then all those are applied in a certain order.

If any of those props or transforms change sourcetype/source/host values, splunk does not go back to the start of the parsing phase to again determine what props and transforms to apply.

It may very well be that you will have to look at splitting the data before it reaches Splunk, such that Splunk can properly apply the relevant config to each different flavor.

DEAD_BEEF
Builder

@FrankVl oh ok, that makes sense. Since the sourcetype is changed by the transforms, it must have already read the TZ props (beforehand) and the foo sourcetype doesn't exist yet so nothing to apply it to. Thanks for the clarification.

0 Karma

FrankVl
Ultra Champion

How are the logs coming in to your indexers? Is there a heavy forwarder involved by any chance?

0 Karma

DEAD_BEEF
Builder

Logs are being sent to a syslog server, UF running on that sending it to the indexers. Splunk TA is then on the indexer doing the magic.

0 Karma

woodcock
Esteemed Legend

The precedence for TZ is:
1: If %z is configured in TIME_FORMAT and is in the event use that EXCEPT...
0: If this matches a TZ_ALIAS setting, override with that.
2: If the forwarder has a TZ setting, use that.
3: If the indexer has a TZ setting, use that.
4: Use the TZ that is configured in the host OS of the indexer (which could be different on every indexer).

You can get some hits as to what has happened for your event by checking the date_zone field.

Much of the time the actual problem is that you are not testing your configuration changes properly so BE ABSOLUTELY SURE that:
1: You restart the splunk service on any device where you deploy a configuration.
2: Add _index_earliest=-1m to your search to make absolutely certain that you are looking ONLY at events that were recently indexed.

DEAD_BEEF
Builder

Hi @woodcock!

1: No TIME_FORMAT
0: No TZ_ALIAS

If I use the TZ that is configured in the indexer (3) or OS (4), wouldn't that set TZ = GMT since both my indexer and the OS is running in GMT? These logs are coming in as CST. I'm sorry I don't understand these suggestions.

I didn't restart the splunk service because when I did --check-restart it said restart is not required since I am only adding a new props.conf.

Tried using _index_earliest=-1m and no results. Neat command!

0 Karma

woodcock
Esteemed Legend

The TZ is set at indextime and is immutable after that. Your TZ configuration changes will never effect already-indexed data (which is why I mentioned that your testing assumptions are probably where the problem is). You must send NEW data in to see if your new settings work.

0 Karma

DEAD_BEEF
Builder

I'm getting new data in constantly. Are you saying that the new data won't have the new props TZ applied to it for some reason?

0 Karma

woodcock
Esteemed Legend

I am saying if something like _index_earliest=-1m and a timepicker of All time returns no events, then you are not getting new events all the time. Did you use All time?

0 Karma

DEAD_BEEF
Builder

@woodcock okay, sorry for my confusion. I ran _index_earliest=-1m for all time and it is showing new events that are "6 hours behind". I refreshed it a few times and verified that the event count is increasing and getting newer timestamped events.

0 Karma

woodcock
Esteemed Legend

Run this command on your indexers:

splunk btool props list --debug | less

It should help pinpoint what value is being used AND FROM WHAT FILE.

0 Karma

DEAD_BEEF
Builder

from running the command I found this in the output:

/$splunk_home$/Splunk_TA_infoblox/local/props.conf   TZ = CST
0 Karma

woodcock
Esteemed Legend

That is the only output for TZ? That is way unexpected.

Are you using INDEXED_EXTRACTIONS or Heavy Forwarders on this input?
If the former, this setting needs to be on the UF, if the latter, this setting needs to be on the HFs.

0 Karma

DEAD_BEEF
Builder

It was the only TZ for that app. there were other hits for TZ but they were all in other apps on the indexer (none in /etc/system/default or /etc/system/local).

This is an indexer with the Splunk TA running. The sourcetype is named via a transforms, and then I am calling that name in props for the TZ fix. No INDEXED_EXTRACTIONS, the data is coming in via .log files from a UF.

0 Karma

woodcock
Esteemed Legend

It doesn't matter what app it is in, it matters what sourcetype it is applied to. In any case, try sending the props.conf to the UF and BE SURE that the sourcetype value for this setting (regardless of what app contains it) matches your data. I suspect that this is your problem.

0 Karma