timestamp recognization for HEC input

mchang_splunk — Wed, 30 Sep 2020 02:52:47 GMT

I'm trying to ingest HEC input into Splunk and set up correct props.conf as below but timestamp is not getting extracted correctly (mainly milliseconds are not matching with index times).

[iks-log4j]
TIME_PREFIX=^
TIME_FORMAT= %Y-%m-%dT%H:%M:%S,%3N
MAX_TIMESTAMP_LOOKAHEAD=23
SHOULD_LINEMERGE=false

LINE_BREAKER=([\r\n]+)

TRUNCATE=999999

Here is the sample event raw data:
2019-09-16T17:54:39,928 10647234 | INFO | XXXXXXXXXXXXXXXXXX

Re: timestamp recognization for HEC input

mchang_splunk — Tue, 05 Nov 2019 20:23:59 GMT

HEC will only extract timestamp from "Time" with epoch time as mentioned in the document:
https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_metadata

For example, you must give the field name "time", otherwise, Splunk will use current time as timestamp:
curl -k "https://:8088/services/collector" -H "Authorization: Splunk " -d '{"host": "test", "index": "main", "sourcetype": "iks-log4j", "event": "2019-09-16T17:54:39,928 whatever.....", "time": 1568656479928}'

Or you can use raw event parsing:
https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Raw_event_parsing

For example, the timestamp will be extracted from the time in "event":

curl https://:8088/services/collector/raw?channel= -H "Authorization: Splunk " -d '{"host": "test", "index": "main", "sourcetype": "iks-log4j", "data": {"event": "2019-09-16T17:54:39,928 whatever....."}'

So EITHER add "time" in event metadata OR use raw event parsing should be able to resolve this issue.

topic timestamp recognization for HEC input in Getting Data In

timestamp recognization for HEC input

LINE_BREAKER=([\r\n]+)

Re: timestamp recognization for HEC input