topic Splunk JSON timestamp in KV_MODE in Getting Data In

Splunk JSON timestamp in KV_MODE

Jakub — Wed, 04 Aug 2021 13:01:28 GMT

Hi All,

I have Event timestamp with miliseconds:

_time with Unix epoch seconds:

and during search the timestamp is from _time, and I would like to have it with milliseconds.

I am using KV_MODE in Search cluster props.conf.

[k8s:dev] KV_MODE = json

and I am trying to do changes in HF props.conf , like TIME_FIELDS, TIME_PREFIX, TIME_FORMAT, but none of them work. INDEXED_EXCTRACTION is turned OFF in HF props.conf

HF props.conf

[k8s:dev] #temporary removed to fix https://jira/browse/DEVA-61153 #INDEXED_EXTRACTIONS = JSON #TIME_PREFIX = {\\"@timestamp\\":\\" TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N TIMESTAMP_FIELDS = @timestamp TRUNCATE = 200000 TRANSFORMS-discard_events = setnull_whitespace_indented,setnull_debug_logging SEDCMD-RemoveLogProp = s/("log":)(.*)(?="stream":)//

this is log, which is coming into the Splunk by HEC.

{"log":"{\"@timestamp\":\"2021-08-03T09:00:57.539+02:00\",\"@version\":\"1\",\"message\":

My question is:

Do changes like TIME_FIELDS, TIME_PREFIX, TIME_FORMAT in HF have effect on this process when INDEXED_EXCTRACTION is not in use?

Thank you very much for your answers.

Re: Splunk JSON timestamp in KV_MODE

kamlesh_vaghela — Wed, 04 Aug 2021 15:45:53 GMT

@Jakub

Can you please try this?

[k8s:dev] [m_json_data_log_1] LINE_BREAKER = ([\r\n]+) MAX_TIMESTAMP_LOOKAHEAD = 29 NO_BINARY_CHECK = true SEDCMD-a = s/\{\"log\":\"//g SEDCMD-b = s/\"\}$//g SEDCMD-c = s/\\"/"/g SHOULD_LINEMERGE = false TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N TIME_PREFIX = {\"log\":\"{\\"@timestamp\\":\\" TRUNCATE = 200000

My Sample Event :

{"log":"{\"@timestamp\":\"2021-08-03T09:00:57.539+02:00\",\"@version\":\"1\",\"message\":\"\"}"}

You can update the SEDCMD Configuration as per your actual event.

Re: Splunk JSON timestamp in KV_MODE

Jakub — Thu, 05 Aug 2021 08:36:48 GMT

Hi @kamlesh_vaghela ,

thank you so much for your answer. But it does not work. I was thinking if it can't be below issue/solution. (I am collecting it via HTTP Event Collector)

Solved: HEC: How to set _time on base of a specific JSON f... - Splunk Community

But my SEDCMD works correctly and I was able to remove "log": property from events, hence the props.conf in HF has to work correctly and its not case of the "event" end point of HEC.

Also SEDCMD is only thing which works.

SEDCMD-RemoveLogProp = s/("log":)(.*)(?="stream":)//

Could it be that INDEXED_EXCTRACTION is OFF and KV_MODE is ON in search?

Thank you very much.