I'm trying to ingest HEC input into Splunk and set up correct props.conf as below but timestamp is not getting extracted correctly (mainly milliseconds are not matching with index times).
[iks-log4j]
TIME_PREFIX=^
TIME_FORMAT= %Y-%m-%dT%H:%M:%S,%3N
MAX_TIMESTAMP_LOOKAHEAD=23
SHOULD_LINEMERGE=false
TRUNCATE=999999
Here is the sample event raw data:
2019-09-16T17:54:39,928 10647234 | INFO | XXXXXXXXXXXXXXXXXX
HEC will only extract timestamp from "Time" with epoch time as mentioned in the document:
https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_met...
For example, you must give the field name "time", otherwise, Splunk will use current time as timestamp:
curl -k "https://:8088/services/collector" -H "Authorization: Splunk " -d '{"host": "test", "index": "main", "sourcetype": "iks-log4j", "event": "2019-09-16T17:54:39,928 whatever.....", "time": 1568656479928}'
Or you can use raw event parsing:
https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Raw_event...
For example, the timestamp will be extracted from the time in "event":
curl https://:8088/services/collector/raw?channel= -H "Authorization: Splunk " -d '{"host": "test", "index": "main", "sourcetype": "iks-log4j", "data": {"event": "2019-09-16T17:54:39,928 whatever....."}'
So EITHER add "time" in event metadata OR use raw event parsing should be able to resolve this issue.
HEC will only extract timestamp from "Time" with epoch time as mentioned in the document:
https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_met...
For example, you must give the field name "time", otherwise, Splunk will use current time as timestamp:
curl -k "https://:8088/services/collector" -H "Authorization: Splunk " -d '{"host": "test", "index": "main", "sourcetype": "iks-log4j", "event": "2019-09-16T17:54:39,928 whatever.....", "time": 1568656479928}'
Or you can use raw event parsing:
https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Raw_event...
For example, the timestamp will be extracted from the time in "event":
curl https://:8088/services/collector/raw?channel= -H "Authorization: Splunk " -d '{"host": "test", "index": "main", "sourcetype": "iks-log4j", "data": {"event": "2019-09-16T17:54:39,928 whatever....."}'
So EITHER add "time" in event metadata OR use raw event parsing should be able to resolve this issue.