Custom month names?

[gljiva]

Hi, I'm having problems timestamp extraction of custom month names (written in my language). Here is a sample log entry:

[26/stu/2010:14:37:40 +0200]

First i tried using TIME_PREFIX and

TIME_FORMAT=%d/%b/%Y:%H:%M%S

but splnuk was unable to understand it and generates wrong timestamp. Next i tried modifying datetime.xml and replacing english month notation with this regex (sij|vel|ozu|tra|svi|lip|srp|kol|ruj|lis|stu|pro) and setting it as argument to DATETIME_CONFIG in props.conf. This also produces wrong timestamp.

What is correct procedure to map english notation (eg. jan feb mar...) to my language (sij, vel, ozu...)? I suppose i need to add transformation and replace my localized names to english ones? How can i do such mapping?

thanks

[gljiva]

Hi, i tried using sed to rewrite month names, it is overwritten fine:

[23/Nov/2010:14:37:40 +0200]

But timestamp is wrong, it seems that splunk first resolves timestamp and then does sed replace.

SEDCMD-sij = s/sij/Jan/
SEDCMD-vel = s/vel/Feb/
SEDCMD-ozu = s/ozu/Mar/
SEDCMD-tra = s/tra/Apr/
SEDCMD-svi = s/svi/May/
SEDCMD-lip = s/lip/Jun/
SEDCMD-srp = s/srp/Jul/
SEDCMD-kol = s/kol/Aug/
SEDCMD-ruj = s/ruj/Sep/
SEDCMD-lis = s/lis/Oct/
SEDCMD-stu = s/stu/Nov/
SEDCMD-pro = s/pro/Dec/
TIME_PREFIX = \d*.\d*.\d*.\d* - \w* \[
TIME_FORMAT = %d/%b/%Y:%H:%M%S

[gljiva]

So how do i rewrite timestamps?

[jrodman]

Timestamps have already been extracted before the events exist. They're part of how we find event boundaries. Thus all event transformations are too late.

[jrodman]

If Splunk is running in the that locale, then I would expect this %b conversion to work. What does the command

# date +'%b'

show for you, in the environment in which splunk is used?

UPDATE: We don't have locale handling in our date parsing at all, currently. I had somehow imagined that we made use of the system library for the specific string decoding, but apparently it is a custom implementation for cross-platform consistency, existence at all on windows, and performance goals.

Essentially this becomes an enhancement request (although a fairly important one) for handling localized european dates. In Asia this hasn't come up (yet) because mostly numerics are used for months, rather than names.

The only short-term workarounds I can recommend are to pre-process the file, or to alter the date format in which it is emitted. Obviously neither is ideal but it's what's possible today.

[jrodman]

There's a defect or a missing step. Please work with splunk support to resolve.

[gljiva]

Incoming datastream is similar to sample provided in first post, every event contains timestamp inside []. I tested locale settings on splunk server indexing data and indeed locale is set to right one (locale matches log locale), but splunk still doesn't understand timestamps. How can i manually rewrite timestamp or tell it real mapping between names?

[jrodman]

What's the incoming datastream like? What hosts are running in a locale where the timestamps will look like this? How does it arrive? The goal here is to have a splunk running in the desired locale, handling that data.

[gljiva]

Hi, Splunk is running in a different locale than one used in logs.

date +'%b' returns
May