Splunk JSON timestamp in KV_MODE

[Jakub]

Hi All,

I have Event timestamp with miliseconds:

_time with Unix epoch seconds:

and during search the timestamp is from _time, and I would like to have it with milliseconds.

I am using KV_MODE in Search cluster props.conf.

[k8s:dev]
KV_MODE = json

and I am trying to do changes in HF props.conf , like TIME_FIELDS, TIME_PREFIX, TIME_FORMAT, but none of them work. INDEXED_EXCTRACTION is turned OFF in HF props.conf

HF props.conf

[k8s:dev]
#temporary removed to fix https://jira/browse/DEVA-61153
#INDEXED_EXTRACTIONS = JSON
#TIME_PREFIX = {\\"@timestamp\\":\\"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N
TIMESTAMP_FIELDS = @timestamp
TRUNCATE = 200000
TRANSFORMS-discard_events = setnull_whitespace_indented,setnull_debug_logging
SEDCMD-RemoveLogProp = s/("log":)(.*)(?="stream":)//

this is log, which is coming into the Splunk by HEC.

{"log":"{\"@timestamp\":\"2021-08-03T09:00:57.539+02:00\",\"@version\":\"1\",\"message\":

My question is:

Do changes like TIME_FIELDS, TIME_PREFIX, TIME_FORMAT in HF have effect on this process when INDEXED_EXCTRACTION is not in use?

Thank you very much for your answers.

[kamlesh_vaghela]

@Jakub 

Can you please try this?

[k8s:dev]
[m_json_data_log_1]
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 29
NO_BINARY_CHECK = true
SEDCMD-a = s/\{\"log\":\"//g
SEDCMD-b = s/\"\}$//g
SEDCMD-c = s/\\"/"/g
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N
TIME_PREFIX = {\"log\":\"{\\"@timestamp\\":\\"
TRUNCATE = 200000

My Sample Event :

{"log":"{\"@timestamp\":\"2021-08-03T09:00:57.539+02:00\",\"@version\":\"1\",\"message\":\"\"}"}

You can update the SEDCMD Configuration as per your actual event.

 

KV

[Jakub]

Hi @kamlesh_vaghela ,

thank you so much for your answer. But it does not work. I was thinking if it can't be below issue/solution. (I am collecting it via HTTP Event Collector)

Solved: HEC: How to set _time on base of a specific JSON f... - Splunk Community

But my SEDCMD works correctly and I was able to remove "log": property from events, hence the props.conf in HF has to work correctly and its not case of the "event" end point of HEC.

Also SEDCMD is only thing which works. 

SEDCMD-RemoveLogProp = s/("log":)(.*)(?="stream":)//

Could it be that INDEXED_EXCTRACTION is OFF and KV_MODE is ON in search?

Thank you very much.