Getting Data In

Please help me identify why Splunk is omitting extracting milliseconds from my JSON

paimonsoror
Builder

Hi folks, running into a strange issue here. Taking the following json:

{   [-] 
     @timestamp:     2018-08-30T02:00:33.993764+00:00       
     level:  info   
     message:    2018-08-30 02:00:33 INFO  Client:54 - Application report for application_1532934978357_294156 (state: RUNNING)
     viaq_msg_id:    MzlhNzc2YjYtOTIzYy00MWY4LWEyMTgtYjc2YmRmZDQ3M2Y0   
}

here it is as raw

{"level":"info","message":"2018-08-30 02:00:33 INFO  Client:54 - Application report for application_1532934978357_294156 (state: RUNNING)","@timestamp":"2018-08-30T02:00:33.993764+00:00","viaq_msg_id":"MzlhNzc2YjYtOTIzYy00MWY4LWEyMTgtYjc2YmRmZDQ3M2Y0",}

The data comes in as a sourcetype of 'fluentd_json' and comes into my HF. I have tried the following as a props.conf:

[fluentd_json]
TIMESTAMP_FIELDS=@timestamp
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%N%Z
INDEXED_EXTRACTIONS=json
KV_MODE=none

but it doesn't seem to work fully. I have tried to use that props on both my indexer cluster as well as my HF (it is also sitting on the SH cluster to prevent duplicate extractions). Both restarted as well. For some reason it omits the milliseconds

Edit:

To clarify what i mean, all of my events from this sourcetype have '.000' for the milliseconds

alt text

Tags (1)
0 Karma

freedomson
Explorer
0 Karma

mstjohn_splunk
Splunk Employee
Splunk Employee

hey @paimonsoror. Thanks for posting.

Would you do me a favor? There is another user that, I believe, is having a similar issue. Would you take a look at their post and verify that for me?

Thanks!

0 Karma

paimonsoror
Builder

@mstjohn_splunk that looks like the exact same issue. Very interesting. A bug you thinking?

0 Karma

mstjohn_splunk
Splunk Employee
Splunk Employee

@paimonsoror, not sure, but thanks for verifying. I'll try to pass this onto the right person so we can find out!

0 Karma

paimonsoror
Builder

@mstjohn_splunk sounds good. Fyi this is in 7.0 . I can try and reproduce in 7.1.x tomorrow

0 Karma

niketn
Legend

@paimonsoror try with %6N for microseconds, also for timezone %:z. Refer to strptime documentation.

TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%6N%:z
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

paimonsoror
Builder

Doesn't seem to work 😞

I even gave this a shot (TIME_PREFIX)

[fluentd_json]
TIMESTAMP_FIELDS=@timestamp
TIME_PREFIX=\"@timestamp\":\"
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%6N%:z
INDEXED_EXTRACTIONS=json
KV_MODE=none

The strange thing is that if I do an 'add data' with a sample json, it works perfectly fine with this:

[ _json ]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json
KV_MODE=none
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%6N%:z
TIMESTAMP_FIELDS=@timestamp

Not sure what the heck is going on. I've doublechecked btool to make sure there are no other props for that sourcetype floating around

And just to be sure, this only really needs to be done on the HF right? Since the HF can cook data, i shouldn't have to also throw this on my indexers.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...