topic Re: Splunk not picking up on timezone set within props in Getting Data In

Splunk not picking up on timezone set within props

DEAD_BEEF — Thu, 17 Jan 2019 22:40:29 GMT

Hi everyone. I have logs that are sent to me in Central Standard Time (-6 hours) but there isn't anything in the TA noting that, so all my logs look like they are 6 hours behind.

As such, I went in and added a props.conf in local with the statement

[infoblox:dhcp]
TZ = CST

Pushed the updated TA through the cluster bundle on my cluster master to all indexers and verified they all received the updated TA. Looking at my latest logs (about 30 afterwards) I still see the latest logs showing up as 6 hours behind (no change). I ran btool to see which props settings were being picked up by the app and indeed it shows it there.

/opt/splunk/bin/splunk cmd btool --app=Splunk_TA_infoblox props list

[infoblox:dhcp]
EVAL-...
EXTRACT-...
TZ = CST

Any ideas? I feel like I'm overlooking something obvious.

Re: Splunk not picking up on timezone set within props

ddrillic — Thu, 17 Jan 2019 23:21:08 GMT

Based on my searching last year - What is the TZ for America Central?

TZ=US/Central seems to be deprecated and it should be TZ=America/Chicago.

Re: Splunk not picking up on timezone set within props

DEAD_BEEF — Fri, 18 Jan 2019 00:56:36 GMT

I changed it to what you have, repushed and it's showing up in btool. Unfortunately, logs are still showing up "6 hours behind" I appreciate the info on the deprecated naming scheme though!

Re: Splunk not picking up on timezone set within props

hdbang_splunk — Fri, 18 Jan 2019 01:39:16 GMT

Try the following format

TZ = CST

Re: Splunk not picking up on timezone set within props

woodcock — Fri, 18 Jan 2019 01:54:23 GMT

The precedence for TZ is:
1: If %z is configured in TIME_FORMAT and is in the event use that EXCEPT...
0: If this matches a TZ_ALIAS setting, override with that.
2: If the forwarder has a TZ setting, use that.
3: If the indexer has a TZ setting, use that.
4: Use the TZ that is configured in the host OS of the indexer (which could be different on every indexer).

You can get some hits as to what has happened for your event by checking the date_zone field.

Much of the time the actual problem is that you are not testing your configuration changes properly so BE ABSOLUTELY SURE that:
1: You restart the splunk service on any device where you deploy a configuration.
2: Add _index_earliest=-1m to your search to make absolutely certain that you are looking ONLY at events that were recently indexed.

Re: Splunk not picking up on timezone set within props

DEAD_BEEF — Fri, 18 Jan 2019 02:14:44 GMT

Hi @woodcock!

1: No TIME_FORMAT
0: No TZ_ALIAS

If I use the TZ that is configured in the indexer (3) or OS (4), wouldn't that set TZ = GMT since both my indexer and the OS is running in GMT? These logs are coming in as CST. I'm sorry I don't understand these suggestions.

I didn't restart the splunk service because when I did --check-restart it said restart is not required since I am only adding a new props.conf.

Tried using _index_earliest=-1m and no results. Neat command!

Re: Splunk not picking up on timezone set within props

woodcock — Fri, 18 Jan 2019 04:18:44 GMT

The TZ is set at indextime and is immutable after that. Your TZ configuration changes will never effect already-indexed data (which is why I mentioned that your testing assumptions are probably where the problem is). You must send NEW data in to see if your new settings work.

Re: Splunk not picking up on timezone set within props

FrankVl — Fri, 18 Jan 2019 07:20:47 GMT

How are the logs coming in to your indexers? Is there a heavy forwarder involved by any chance?

Re: Splunk not picking up on timezone set within props

DEAD_BEEF — Fri, 18 Jan 2019 13:49:01 GMT

Logs are being sent to a syslog server, UF running on that sending it to the indexers. Splunk TA is then on the indexer doing the magic.

Re: Splunk not picking up on timezone set within props

DEAD_BEEF — Fri, 18 Jan 2019 14:11:50 GMT

I'm getting new data in constantly. Are you saying that the new data won't have the new props TZ applied to it for some reason?

Re: Splunk not picking up on timezone set within props

woodcock — Fri, 18 Jan 2019 15:23:53 GMT

I am saying if something like _index_earliest=-1m and a timepicker of All time returns no events, then you are not getting new events all the time. Did you use All time?

Re: Splunk not picking up on timezone set within props

woodcock — Fri, 18 Jan 2019 15:24:22 GMT

What he is using should work, but I agree with this.

Re: Splunk not picking up on timezone set within props

DEAD_BEEF — Fri, 18 Jan 2019 15:33:01 GMT

Updated it to your suggestions, but events are still showing up as 6 hours behind. I haven't done a rolling restart as --check-restart said it is not required, but at this point I'm not sure what else to try.

Re: Splunk not picking up on timezone set within props

DEAD_BEEF — Fri, 18 Jan 2019 15:36:10 GMT

@woodcock okay, sorry for my confusion. I ran _index_earliest=-1m for all time and it is showing new events that are "6 hours behind". I refreshed it a few times and verified that the event count is increasing and getting newer timestamped events.

Re: Splunk not picking up on timezone set within props

woodcock — Fri, 18 Jan 2019 16:20:09 GMT

Run this command on your indexers:

splunk btool props list --debug | less

It should help pinpoint what value is being used AND FROM WHAT FILE.

Re: Splunk not picking up on timezone set within props

DEAD_BEEF — Fri, 18 Jan 2019 16:37:29 GMT

from running the command I found this in the output:

/$splunk_home$/Splunk_TA_infoblox/local/props.conf   TZ = CST

Re: Splunk not picking up on timezone set within props

woodcock — Fri, 18 Jan 2019 16:45:50 GMT

That is the only output for TZ? That is way unexpected.

Are you using INDEXED_EXTRACTIONS or Heavy Forwarders on this input?
If the former, this setting needs to be on the UF, if the latter, this setting needs to be on the HFs.

Re: Splunk not picking up on timezone set within props

DEAD_BEEF — Fri, 18 Jan 2019 17:27:12 GMT

It was the only TZ for that app. there were other hits for TZ but they were all in other apps on the indexer (none in /etc/system/default or /etc/system/local).

This is an indexer with the Splunk TA running. The sourcetype is named via a transforms, and then I am calling that name in props for the TZ fix. No INDEXED_EXTRACTIONS, the data is coming in via .log files from a UF.

Re: Splunk not picking up on timezone set within props

woodcock — Fri, 18 Jan 2019 17:30:36 GMT

It doesn't matter what app it is in, it matters what sourcetype it is applied to. In any case, try sending the props.conf to the UF and BE SURE that the sourcetype value for this setting (regardless of what app contains it) matches your data. I suspect that this is your problem.

Re: Splunk not picking up on timezone set within props

arkadyz1 — Fri, 18 Jan 2019 18:08:08 GMT

At this point, I would go over to the universal forwarders and check how the timestamp is generated (is it in the . format?) and any timezone settings there, including timezone setting of the OS.