Timestamp milliseconds not appearing

[mwdbhyat]

Hi there,

I am extracting a timestamp in props.. everything is working fine except for the milliseconds at the end of it.

Date format is 2017-07-11 08:54:12,815 -- my extraction is %Y-%m-%d %H-%M-%S,%3N

That extraction works for similar data with a .(period) before the milliseconds instead of a comma - changing the end of the extract to .%3N - but it never works for a comma.

Is there anything else needed when trying to extract milliseconds after a comma?

Thanks

[freedomson]

Please take a look into:
https://answers.splunk.com/answers/688698/why-are-milliseconds-not-being-parsed-in-cluster-e.html

[niketn · Accepted solution]

This seems to work for me as well. Can you try createing a new sourcetype with only TIME_FORMAT and MAX_TIMESTAMP_LOOKAHEAD? Test with your data in preview mode whether it is working as expected or not.

[customSourcetypename]
DATETIME_CONFIG = 
MAX_TIMESTAMP_LOOKAHEAD = 24
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
category = Custom
pulldown_type = true

[gcusello]

Hi mwdbhyat,
let me understand:
you have another sourcetype with time format %Y-%m-%d %H-%M-%S.%3N that works with other data and to read these data, do you want to use the same time format or to create another one?

If you want to create another one it's easy because you insert the correct format in props.conf
TIME_FORMAT = %Y-%m-%d %H-%M-%S,%3N

If instead you want to use the same sourcetype, don't set any time format, Splunk recognize both the time format (I tried it).

Bye.
Giuseppe

[mwdbhyat]

Hi Giuseppe,

The issue im facing is that it will not extract the milliseconds from the timestamp when there is a comma in the timestamp before the milliseconds..

[niketn]

@cusello, slight correction to your answer, Hour Minute and Second are separated by colons(:) not hyphens (-).
@mwdbhyat, please try out colon as separator for Time. I have added a run-anywhere search to test.

| makeresults
| eval epochTime=strptime("2017-07-11 08:54:12,815","%Y-%m-%d %H:%M:%S,%3N")
| eval stringTime=strptime(epochTime,"%Y-%m-%d %H:%M:%S,%3N")
| table epochTime stringTime

[mwdbhyat]

Hi guys,

Thanks for the help. @niketnilay - that was a typo in this question 🙂 ..I used colons not hyphens. the makeresults query only returns epochTime value?

The issue at hand is this - %Y-%m-%d %H-%M-%S,%3N should match 2017-07-11 08:54:12,815 but it doesnt take the milliseconds.. Thats the only problem..

Not setting a TIME_FORMAT will be less performant so I dont want to leave it blank.

[niketn]

makeresults is a generating command which allows you to create your own variable/data to be displayed.

strptime() converts string time to epoch and strftime() converts from epoch to string. If you compare the two values (epochTime and stringTime) you will see that 815 is present in both hence milliseconds is working as expected.

Is this extraction being done while sourcetype definition in props.conf? Can you add your current settings and some sample raw data(mock or anonymize sensitive info where ever needed?

[mwdbhyat]

Ah right i get it now(think you had a typo in your command strp instead of strf for the stringTime.

[sourcectypename]
LINE_BREAKER = ([\r\n]+)
TIME_FORMAT = as above
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 27 ( i overset this this, initially it matched the actual length..thought it might be cutting it off)
Truncate 999999
SHOULD_LINEMERGE = false

sample data:

2017-07-11 08:54:12,815 31 [INFO] - - 5ms textHere MoreTEXT here[652].moretextHere(): MoreTextHere

[gcusello]

Hi mwdbhyat,
if you set the time format (with dot or comma), it's correct that Splunk doesn't recognize both timestamps but only ther one that matches your settings.
If instead you don't set the time format and leave that Splunk recognizes time format, it recognizes both the time formats.
I created a file with your two time formats and Splunk recognized both of them.
Bye.
Giuseppe