Please help me identify why Splunk is omitting extracting milliseconds from my JSON

[paimonsoror]

Hi folks, running into a strange issue here. Taking the following json:

{   [-] 
     @timestamp:     2018-08-30T02:00:33.993764+00:00       
     level:  info   
     message:    2018-08-30 02:00:33 INFO  Client:54 - Application report for application_1532934978357_294156 (state: RUNNING)
     viaq_msg_id:    MzlhNzc2YjYtOTIzYy00MWY4LWEyMTgtYjc2YmRmZDQ3M2Y0   
}

here it is as raw

{"level":"info","message":"2018-08-30 02:00:33 INFO  Client:54 - Application report for application_1532934978357_294156 (state: RUNNING)","@timestamp":"2018-08-30T02:00:33.993764+00:00","viaq_msg_id":"MzlhNzc2YjYtOTIzYy00MWY4LWEyMTgtYjc2YmRmZDQ3M2Y0",}

The data comes in as a sourcetype of 'fluentd_json' and comes into my HF. I have tried the following as a props.conf:

[fluentd_json]
TIMESTAMP_FIELDS=@timestamp
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%N%Z
INDEXED_EXTRACTIONS=json
KV_MODE=none

but it doesn't seem to work fully. I have tried to use that props on both my indexer cluster as well as my HF (it is also sitting on the SH cluster to prevent duplicate extractions). Both restarted as well. For some reason it omits the milliseconds

Edit:

To clarify what i mean, all of my events from this sourcetype have '.000' for the milliseconds

[freedomson]

Please take a look into:
https://answers.splunk.com/answers/688698/why-are-milliseconds-not-being-parsed-in-cluster-e.html

[mstjohn_splunk]

hey @paimonsoror. Thanks for posting.

Would you do me a favor? There is another user that, I believe, is having a similar issue. Would you take a look at their post and verify that for me?

Thanks!

[paimonsoror]

@mstjohn_splunk that looks like the exact same issue. Very interesting. A bug you thinking?

[mstjohn_splunk]

@paimonsoror, not sure, but thanks for verifying. I'll try to pass this onto the right person so we can find out!

[paimonsoror]

@mstjohn_splunk sounds good. Fyi this is in 7.0 . I can try and reproduce in 7.1.x tomorrow

[niketn]

@paimonsoror try with %6N for microseconds, also for timezone %:z. Refer to strptime documentation.

TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%6N%:z

[paimonsoror]

Doesn't seem to work 😞

I even gave this a shot (TIME_PREFIX)

[fluentd_json]
TIMESTAMP_FIELDS=@timestamp
TIME_PREFIX=\"@timestamp\":\"
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%6N%:z
INDEXED_EXTRACTIONS=json
KV_MODE=none

The strange thing is that if I do an 'add data' with a sample json, it works perfectly fine with this:

[ _json ]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json
KV_MODE=none
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%6N%:z
TIMESTAMP_FIELDS=@timestamp

Not sure what the heck is going on. I've doublechecked btool to make sure there are no other props for that sourcetype floating around

And just to be sure, this only really needs to be done on the HF right? Since the HF can cook data, i shouldn't have to also throw this on my indexers.