Custom month names?

[gljiva]

Hi, I'm having problems timestamp extraction of custom month names (written in my language). Here is a sample log entry:

[26/stu/2010:14:37:40 +0200]

First i tried using TIME_PREFIX and

TIME_FORMAT=%d/%b/%Y:%H:%M%S

but splnuk was unable to understand it and generates wrong timestamp. Next i tried modifying datetime.xml and replacing english month notation with this regex (sij|vel|ozu|tra|svi|lip|srp|kol|ruj|lis|stu|pro) and setting it as argument to DATETIME_CONFIG in props.conf. This also produces wrong timestamp.

What is correct procedure to map english notation (eg. jan feb mar...) to my language (sij, vel, ozu...)? I suppose i need to add transformation and replace my localized names to english ones? How can i do such mapping?

thanks

[pellegrini]

A very old post but still relevant if the log timestamp format cannot be change.

If exact timestamp is not needed I would set this to current or none for the specific sourcetype in props.conf. It is a very quick fix.

DATETIME_CONFIG = [CURRENT | NONE]

 Alternative just extract the time and not the date. Works fine as long as the events are indexed the same day as they are written.

TIME_FORMAT = %H:%M:%S

As last alternative datetime.xml is possible but maybe not so easy. In this case it might be enough to modify the "litmonth" attributes. Just remember to copy the file and rename it to something else and use that modified file for this specific sourcetype only. Modifying datetime.xml would impact all transactions on so do not do that.

[gljiva]

Hi, i tried using sed to rewrite month names, it is overwritten fine:

[23/Nov/2010:14:37:40 +0200]

But timestamp is wrong, it seems that splunk first resolves timestamp and then does sed replace.

SEDCMD-sij = s/sij/Jan/
SEDCMD-vel = s/vel/Feb/
SEDCMD-ozu = s/ozu/Mar/
SEDCMD-tra = s/tra/Apr/
SEDCMD-svi = s/svi/May/
SEDCMD-lip = s/lip/Jun/
SEDCMD-srp = s/srp/Jul/
SEDCMD-kol = s/kol/Aug/
SEDCMD-ruj = s/ruj/Sep/
SEDCMD-lis = s/lis/Oct/
SEDCMD-stu = s/stu/Nov/
SEDCMD-pro = s/pro/Dec/
TIME_PREFIX = \d*.\d*.\d*.\d* - \w* \[
TIME_FORMAT = %d/%b/%Y:%H:%M%S

[gljiva]

So how do i rewrite timestamps?

[jrodman]

Timestamps have already been extracted before the events exist. They're part of how we find event boundaries. Thus all event transformations are too late.

[jrodman]

If Splunk is running in the that locale, then I would expect this %b conversion to work. What does the command

# date +'%b'

show for you, in the environment in which splunk is used?

UPDATE: We don't have locale handling in our date parsing at all, currently. I had somehow imagined that we made use of the system library for the specific string decoding, but apparently it is a custom implementation for cross-platform consistency, existence at all on windows, and performance goals.

Essentially this becomes an enhancement request (although a fairly important one) for handling localized european dates. In Asia this hasn't come up (yet) because mostly numerics are used for months, rather than names.

The only short-term workarounds I can recommend are to pre-process the file, or to alter the date format in which it is emitted. Obviously neither is ideal but it's what's possible today.

[jrodman]

There's a defect or a missing step. Please work with splunk support to resolve.

[gljiva]

Incoming datastream is similar to sample provided in first post, every event contains timestamp inside []. I tested locale settings on splunk server indexing data and indeed locale is set to right one (locale matches log locale), but splunk still doesn't understand timestamps. How can i manually rewrite timestamp or tell it real mapping between names?

[jrodman]

What's the incoming datastream like? What hosts are running in a locale where the timestamps will look like this? How does it arrive? The goal here is to have a splunk running in the desired locale, handling that data.

[gljiva]

Hi, Splunk is running in a different locale than one used in logs.

date +'%b' returns
May