Getting Data In
Highlighted

Special epoch timestamp recognition

Motivator

Can Splunk somehow recognize the following timestamp format: 1.375944219123E9

It is the epoch time written in float exponential notation with milliseconds

I did not see that strptime supports this format

I then thought that I could just get rid of the "." using SEDCMD in props.conf but it is only executed after the timestamp recognition

Aaand I think it is not possible to write a custom datetime.xml that somehow drops the "." because it is regex based and you can't just drop the "." in a capturing group, but I might be wrong.

Any Ideas? Also a confirmation that it is not possible to read this format would help.

Thanks

Chris

0 Karma
Highlighted

Re: Special epoch timestamp recognition

Ultra Champion

Hmm. Tricky. Is it possible to make a TRANSFORM on _time?

It would require that you set your TIME_PREFIX to include the "1."-part of the timestamp. And then set the TIME_FORMAT to %s%3N. That would give you a timestamp in early 1980's.

Then with a TRANSFORM on _time you add the starting "1" (and perhaps remove the millisecond part)...

I have not done this before, so take it as a theory that might be worth investigating. Perhaps this is not at all possible. It's at times like these I wish I knew all of the parsing/indexing phase processors by heart, and in which order they come. 🙂

From the docs on transforms.conf:

FORMAT = <string>
* NOTE: This option is valid for both index-time and search-time field extraction. However, FORMAT 
  behaves differently depending on whether the extraction is performed at index time or 
  search time.
* This attribute specifies the format of the event, including any field names or values you want 
  to add.
* FORMAT for index-time extractions:
    * Use $n (for example $1, $2, etc) to specify the output of each REGEX match. 
    * If REGEX does not have n groups, the matching fails. 
    * The special identifier $0 represents what was in the DEST_KEY before the REGEX was performed.
    * At index time only, you can use FORMAT to create concatenated fields:
        * FORMAT = ipaddress::$1.$2.$3.$4
    * When you create concatenated fields with FORMAT, "$" is the only special character. It is 
      treated as a prefix for regex-capturing groups only if it is followed by a number and only 
      if the number applies to an existing capturing group. So if REGEX has only one capturing 
      group and its value is "bar", then:
        * "FORMAT = foo$1" yields "foobar"
        * "FORMAT = foo$bar" yields "foo$bar"
        * "FORMAT = foo$1234" yields "foo$1234"
        * "FORMAT = foo$1\$2" yields "foobar\$2"
    * At index-time, FORMAT defaults to <stanza-name>::$1

Hope this helps a little,

K


Update:

Hi K,
you sent me in the right direction. This works without the subsecond part:

props.conf
[epo]
TRANSFORMS-epo=epo

transforms.conf
[epo]
DEST_KEY = _time
#This doesn't work
#REGEX =  (1)\.(\d{12})E9
#This does work
REGEX =  (1)\.(\d{9})\d{3}E9
FORMAT = $1$2

View solution in original post

Highlighted

Re: Special epoch timestamp recognition

Ultra Champion

glad to hear that it worked...

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.