topic Re: Index forwarder and time_format settings in Getting Data In

Index forwarder and time_format settings

rchan11 — Mon, 28 Sep 2020 14:55:27 GMT

Hi,

I have index forwarders forwarding information to a centralized splunk server. However, the timestamps are being parsed incorrectly. Does the C:\Program Files\Splunk\etc\system\local\props.conf file have to be updated on the source systems or the server hosting the splunk searches?

My date format is 2012/07/26:07:44:35.696 PDT
and it's recognizing incorrectly with timestamp format %Y/%m/%d:%H:%M:%S

Not sure why with the following line:
Date/Time Started: 2012/08/02:05:53:30.486 CST Type/Server/Application/Database: Essbase/stnsvpessdb/SLS_TD/TONS Business Rule Name: BR189 By HBR user: HypBatch By Essbase user: admin Values entered for run-time prompts: [Variable] rtp_Product:"All_Products"

It does not parse the time correctly with the following settings:
# your settings
MAX_TIMESTAMP_LOOKAHEAD=23
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y/%m/%d:%H:%M:%S
TIME_PREFIX=^Date/Time Started:\t|^Date/Time Ended:\t

Splunk is recognizing the above time as 2 AM instead of 5 AM. Not sure why it's using the 2 digit twice.

Thanks,
Ryan

Re: Index forwarder and time_format settings

kristian_kolb — Tue, 08 Oct 2013 19:48:16 GMT

TIME_FORMAT is a valid props.conf configuration attribute that is applied in the Parsing Phase. Depending on your setup that can be on a Heavy Forwarder or an Indexer. It does not matter from where the searches are being made. See below:

http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings

Re: Index forwarder and time_format settings

rchan11 — Tue, 08 Oct 2013 20:12:13 GMT

Thanks K, I found out that I can do the settings on the input file however I can't get Splunk to recognize my formatting correctly now.

Re: Index forwarder and time_format settings

kristian_kolb — Tue, 08 Oct 2013 20:22:31 GMT

Hi. The TIME_FORMAT might require a little bit of tweaking, I'd suggest the following;

TIME_FORMAT = %Y/%m/%d:%H:%M:%S.%3N %Z MAX_TIMESTAMP_LOOKAHEAD = 50

You probably do not need any TIME_PREFIX, but if you think you do, just try with;

TIME_PREFIX = :\s+

http://docs.splunk.com/Documentation/Splunk/6.0/SearchReference/Commontimeformatvariables

EDIT:typo

Re: Index forwarder and time_format settings

kristian_kolb — Tue, 08 Oct 2013 20:40:18 GMT

Looks like different time zones. See TIME_FORMAT in answer below.

PDT=Pacific Daylight Saving Time??
CST=Central Standard Time??

Could that be the case?

Re: Index forwarder and time_format settings

rchan11 — Tue, 08 Oct 2013 20:43:42 GMT

Nope I looked and that wasn't it. I changed the formatting according to your suggestion above and the preview worked. The formatting worked on the forwarding server preview of the document, but the central splunk server still sees it incorrectly.

Re: Index forwarder and time_format settings

kristian_kolb — Tue, 08 Oct 2013 20:50:09 GMT

Not sure I follow your topology. Do you have UF -> Indexer -> Indexer, i.e. and index-and-forward scenario?

In that case, the config should be on the first indexer.

If you have a Heavy Forwarder, like HF -> Indexer, or UF -> HF -> Indexer, the config should be on the HF.

And of course, the setting will only apply to new events coming in.

Re: Index forwarder and time_format settings

rchan11 — Tue, 08 Oct 2013 20:58:35 GMT

I believe I have the Heavy Forwarder that does the indexing and forwards it to the central search-enabled splunk server (using the same index name).

I made my formatting adjustments on the Heavy Forwarding (first indexer) server but the centralized server still sees incorrect timestamps. I'm assuming the first indexer handles the formatting and forwards the data correctly to the central splunk server which handles the searching. New events have happened and there is no difference between the old and new timestamp recognition despite changing it on the first indexer.

Re: Index forwarder and time_format settings

kristian_kolb — Tue, 08 Oct 2013 21:09:54 GMT

Normally when using a HF (with or without local indexing) it will send 'cooked' data, i.e. it has no need of further parsing. However if you are sending the data as e.g. syslog routing, it is not in Splunk format, and should be parsed again... but this sounds weird. Did you go out of your way to set up the forwarding, or did you use 'standard' methods?

Re: Index forwarder and time_format settings

rchan11 — Tue, 08 Oct 2013 21:19:32 GMT

I used HF with local indexing and have the receiving search server with the same index. It's set to forward updates to the logs as updates are being done to the log. I used the out-of-the-box method to set up forwarding. There is no customizations being done to the forwarding of the data.

Re: Index forwarder and time_format settings

kristian_kolb — Tue, 08 Oct 2013 21:29:43 GMT

did you restart the HF? I see that you say 'worked in preview'. This type of operation is probably going to need a restart, or that you go to

http://your_hf/en-US/debug/refresh

which may be enough to read the new config.

Re: Index forwarder and time_format settings

rchan11 — Tue, 08 Oct 2013 21:32:31 GMT

Ah, I have not restarted the server yet. Restarting....Appreciate the tips, Kristian!

Re: Index forwarder and time_format settings

kristian_kolb — Tue, 08 Oct 2013 21:42:03 GMT

you're welcome. did it work?

Re: Index forwarder and time_format settings

rchan11 — Tue, 08 Oct 2013 21:44:45 GMT

Just checked it out and nope. Unfortunately that did not work. The search server does see the new sourceType I assigned to the updates with the corrected time format, however the search server still reports all today's transactions as occurring at 8am instead of on Oct 8th in the afternoon.

Re: Index forwarder and time_format settings

kristian_kolb — Tue, 08 Oct 2013 21:48:51 GMT

and if you search for them on the HF, they look alright?

Re: Index forwarder and time_format settings

rchan11 — Wed, 09 Oct 2013 16:49:22 GMT

Finally got it working! I upgraded to version 6. I guess the receiving server didn't recognize the new sourcetype with the new time format.

Thanks for your help Kristian!