Index forwarder and time_format settings

[rchan11]

Hi,

I have index forwarders forwarding information to a centralized splunk server. However, the timestamps are being parsed incorrectly. Does the C:\Program Files\Splunk\etc\system\local\props.conf file have to be updated on the source systems or the server hosting the splunk searches?

My date format is 2012/07/26:07:44:35.696 PDT
and it's recognizing incorrectly with timestamp format %Y/%m/%d:%H:%M:%S

Not sure why with the following line:
Date/Time Started: 2012/08/02:05:53:30.486 CST Type/Server/Application/Database: Essbase/stnsvpessdb/SLS_TD/TONS Business Rule Name: BR189 By HBR user: HypBatch By Essbase user: admin Values entered for run-time prompts: [Variable] rtp_Product:"All_Products"

It does not parse the time correctly with the following settings:
# your settings
MAX_TIMESTAMP_LOOKAHEAD=23
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y/%m/%d:%H:%M:%S
TIME_PREFIX=^Date/Time Started:\t|^Date/Time Ended:\t

Splunk is recognizing the above time as 2 AM instead of 5 AM. Not sure why it's using the 2 digit twice.

Thanks,
Ryan

[rchan11]

Finally got it working! I upgraded to version 6. I guess the receiving server didn't recognize the new sourcetype with the new time format.

Thanks for your help Kristian!

[kristian_kolb]

and if you search for them on the HF, they look alright?

[rchan11]

Just checked it out and nope. Unfortunately that did not work. The search server does see the new sourceType I assigned to the updates with the corrected time format, however the search server still reports all today's transactions as occurring at 8am instead of on Oct 8th in the afternoon.

[kristian_kolb]

you're welcome. did it work?

[rchan11]

Ah, I have not restarted the server yet. Restarting....Appreciate the tips, Kristian!

[kristian_kolb]

did you restart the HF? I see that you say 'worked in preview'. This type of operation is probably going to need a restart, or that you go to

http://your_hf/en-US/debug/refresh

which may be enough to read the new config.

[rchan11]

I used HF with local indexing and have the receiving search server with the same index. It's set to forward updates to the logs as updates are being done to the log. I used the out-of-the-box method to set up forwarding. There is no customizations being done to the forwarding of the data.

[kristian_kolb]

Normally when using a HF (with or without local indexing) it will send 'cooked' data, i.e. it has no need of further parsing. However if you are sending the data as e.g. syslog routing, it is not in Splunk format, and should be parsed again... but this sounds weird. Did you go out of your way to set up the forwarding, or did you use 'standard' methods?

[rchan11]

I believe I have the Heavy Forwarder that does the indexing and forwards it to the central search-enabled splunk server (using the same index name).

I made my formatting adjustments on the Heavy Forwarding (first indexer) server but the centralized server still sees incorrect timestamps. I'm assuming the first indexer handles the formatting and forwards the data correctly to the central splunk server which handles the searching. New events have happened and there is no difference between the old and new timestamp recognition despite changing it on the first indexer.

[kristian_kolb]

Not sure I follow your topology. Do you have UF -> Indexer -> Indexer, i.e. and index-and-forward scenario?

In that case, the config should be on the first indexer.

If you have a Heavy Forwarder, like HF -> Indexer, or UF -> HF -> Indexer, the config should be on the HF.

And of course, the setting will only apply to new events coming in.

[rchan11]

Nope I looked and that wasn't it. I changed the formatting according to your suggestion above and the preview worked. The formatting worked on the forwarding server preview of the document, but the central splunk server still sees it incorrectly.

[kristian_kolb]

Looks like different time zones. See TIME_FORMAT in answer below.

PDT=Pacific Daylight Saving Time??
CST=Central Standard Time??

Could that be the case?

[kristian_kolb · Accepted solution]

TIME_FORMAT is a valid props.conf configuration attribute that is applied in the Parsing Phase. Depending on your setup that can be on a Heavy Forwarder or an Indexer. It does not matter from where the searches are being made. See below:

http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings

/K

[kristian_kolb]

Hi. The TIME_FORMAT might require a little bit of tweaking, I'd suggest the following;

TIME_FORMAT = %Y/%m/%d:%H:%M:%S.%3N %Z
MAX_TIMESTAMP_LOOKAHEAD = 50

You probably do not need any TIME_PREFIX, but if you think you do, just try with;

TIME_PREFIX = :\s+

http://docs.splunk.com/Documentation/Splunk/6.0/SearchReference/Commontimeformatvariables

EDIT:typo

/K

[rchan11]

Thanks K, I found out that I can do the settings on the input file however I can't get Splunk to recognize my formatting correctly now.