Getting Data In

Please help me identify why Splunk is omitting extracting milliseconds from my JSON

paimonsoror
Builder

Hi folks, running into a strange issue here. Taking the following json:

{   [-] 
     @timestamp:     2018-08-30T02:00:33.993764+00:00       
     level:  info   
     message:    2018-08-30 02:00:33 INFO  Client:54 - Application report for application_1532934978357_294156 (state: RUNNING)
     viaq_msg_id:    MzlhNzc2YjYtOTIzYy00MWY4LWEyMTgtYjc2YmRmZDQ3M2Y0   
}

here it is as raw

{"level":"info","message":"2018-08-30 02:00:33 INFO  Client:54 - Application report for application_1532934978357_294156 (state: RUNNING)","@timestamp":"2018-08-30T02:00:33.993764+00:00","viaq_msg_id":"MzlhNzc2YjYtOTIzYy00MWY4LWEyMTgtYjc2YmRmZDQ3M2Y0",}

The data comes in as a sourcetype of 'fluentd_json' and comes into my HF. I have tried the following as a props.conf:

[fluentd_json]
TIMESTAMP_FIELDS=@timestamp
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%N%Z
INDEXED_EXTRACTIONS=json
KV_MODE=none

but it doesn't seem to work fully. I have tried to use that props on both my indexer cluster as well as my HF (it is also sitting on the SH cluster to prevent duplicate extractions). Both restarted as well. For some reason it omits the milliseconds

Edit:

To clarify what i mean, all of my events from this sourcetype have '.000' for the milliseconds

alt text

Tags (1)
0 Karma

freedomson
Explorer
0 Karma

mstjohn_splunk
Splunk Employee
Splunk Employee

hey @paimonsoror. Thanks for posting.

Would you do me a favor? There is another user that, I believe, is having a similar issue. Would you take a look at their post and verify that for me?

Thanks!

0 Karma

paimonsoror
Builder

@mstjohn_splunk that looks like the exact same issue. Very interesting. A bug you thinking?

0 Karma

mstjohn_splunk
Splunk Employee
Splunk Employee

@paimonsoror, not sure, but thanks for verifying. I'll try to pass this onto the right person so we can find out!

0 Karma

paimonsoror
Builder

@mstjohn_splunk sounds good. Fyi this is in 7.0 . I can try and reproduce in 7.1.x tomorrow

0 Karma

niketnilay
Legend

@paimonsoror try with %6N for microseconds, also for timezone %:z. Refer to strptime documentation.

TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%6N%:z
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

paimonsoror
Builder

Doesn't seem to work 😞

I even gave this a shot (TIME_PREFIX)

[fluentd_json]
TIMESTAMP_FIELDS=@timestamp
TIME_PREFIX=\"@timestamp\":\"
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%6N%:z
INDEXED_EXTRACTIONS=json
KV_MODE=none

The strange thing is that if I do an 'add data' with a sample json, it works perfectly fine with this:

[ _json ]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json
KV_MODE=none
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%6N%:z
TIMESTAMP_FIELDS=@timestamp

Not sure what the heck is going on. I've doublechecked btool to make sure there are no other props for that sourcetype floating around

And just to be sure, this only really needs to be done on the HF right? Since the HF can cook data, i shouldn't have to also throw this on my indexers.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!