topic Re: Custom month names? in Getting Data In

Custom month names?

gljiva — Wed, 05 May 2010 04:58:55 GMT

Hi, I'm having problems timestamp extraction of custom month names (written in my language). Here is a sample log entry:

[26/stu/2010:14:37:40 +0200]

First i tried using TIME_PREFIX and

TIME_FORMAT=%d/%b/%Y:%H:%M%S

but splnuk was unable to understand it and generates wrong timestamp. Next i tried modifying datetime.xml and replacing english month notation with this regex (sij|vel|ozu|tra|svi|lip|srp|kol|ruj|lis|stu|pro) and setting it as argument to DATETIME_CONFIG in props.conf. This also produces wrong timestamp.

What is correct procedure to map english notation (eg. jan feb mar...) to my language (sij, vel, ozu...)? I suppose i need to add transformation and replace my localized names to english ones? How can i do such mapping?

thanks

Re: Custom month names?

jrodman — Wed, 05 May 2010 12:55:16 GMT

If Splunk is running in the that locale, then I would expect this %b conversion to work. What does the command

# date +'%b'

show for you, in the environment in which splunk is used?

UPDATE: We don't have locale handling in our date parsing at all, currently. I had somehow imagined that we made use of the system library for the specific string decoding, but apparently it is a custom implementation for cross-platform consistency, existence at all on windows, and performance goals.

Essentially this becomes an enhancement request (although a fairly important one) for handling localized european dates. In Asia this hasn't come up (yet) because mostly numerics are used for months, rather than names.

The only short-term workarounds I can recommend are to pre-process the file, or to alter the date format in which it is emitted. Obviously neither is ideal but it's what's possible today.

Re: Custom month names?

gljiva — Wed, 05 May 2010 15:02:25 GMT

Hi, Splunk is running in a different locale than one used in logs.

date +'%b' returns
May

Re: Custom month names?

gljiva — Wed, 05 May 2010 15:23:04 GMT

Hi, i tried using sed to rewrite month names, it is overwritten fine:

[23/Nov/2010:14:37:40 +0200]

But timestamp is wrong, it seems that splunk first resolves timestamp and then does sed replace.

SEDCMD-sij = s/sij/Jan/
SEDCMD-vel = s/vel/Feb/
SEDCMD-ozu = s/ozu/Mar/
SEDCMD-tra = s/tra/Apr/
SEDCMD-svi = s/svi/May/
SEDCMD-lip = s/lip/Jun/
SEDCMD-srp = s/srp/Jul/
SEDCMD-kol = s/kol/Aug/
SEDCMD-ruj = s/ruj/Sep/
SEDCMD-lis = s/lis/Oct/
SEDCMD-stu = s/stu/Nov/
SEDCMD-pro = s/pro/Dec/
TIME_PREFIX = \d*.\d*.\d*.\d* - \w* \[
TIME_FORMAT = %d/%b/%Y:%H:%M%S

Re: Custom month names?

jrodman — Wed, 05 May 2010 21:24:40 GMT

What's the incoming datastream like? What hosts are running in a locale where the timestamps will look like this? How does it arrive? The goal here is to have a splunk running in the desired locale, handling that data.

Re: Custom month names?

jrodman — Wed, 05 May 2010 21:26:59 GMT

Timestamps have already been extracted before the events exist. They're part of how we find event boundaries. Thus all event transformations are too late.

Re: Custom month names?

gljiva — Thu, 06 May 2010 00:14:58 GMT

So how do i rewrite timestamps?

Re: Custom month names?

gljiva — Thu, 06 May 2010 00:17:13 GMT

Incoming datastream is similar to sample provided in first post, every event contains timestamp inside []. I tested locale settings on splunk server indexing data and indeed locale is set to right one (locale matches log locale), but splunk still doesn't understand timestamps. How can i manually rewrite timestamp or tell it real mapping between names?

Re: Custom month names?

jrodman — Sat, 08 May 2010 00:59:34 GMT

There's a defect or a missing step. Please work with splunk support to resolve.

Re: Custom month names?

pellegrini — Wed, 22 May 2024 13:15:13 GMT

A very old post but still relevant if the log timestamp format cannot be change.

If exact timestamp is not needed I would set this to current or none for the specific sourcetype in props.conf. It is a very quick fix.

DATETIME_CONFIG = [CURRENT | NONE]

Alternative just extract the time and not the date. Works fine as long as the events are indexed the same day as they are written.

TIME_FORMAT = %H:%M:%S

As last alternative datetime.xml is possible but maybe not so easy. In this case it might be enough to modify the "litmonth" attributes. Just remember to copy the file and rename it to something else and use that modified file for this specific sourcetype only. Modifying datetime.xml would impact all transactions on so do not do that.