Getting Data In

timestamp recognization for HEC input

mchang_splunk
Splunk Employee
Splunk Employee

I'm trying to ingest HEC input into Splunk and set up correct props.conf as below but timestamp is not getting extracted correctly (mainly milliseconds are not matching with index times).

[iks-log4j]
TIME_PREFIX=^
TIME_FORMAT= %Y-%m-%dT%H:%M:%S,%3N
MAX_TIMESTAMP_LOOKAHEAD=23
SHOULD_LINEMERGE=false

LINE_BREAKER=([\r\n]+)

TRUNCATE=999999

Here is the sample event raw data:
2019-09-16T17:54:39,928 10647234 | INFO | XXXXXXXXXXXXXXXXXX

Tags (1)
0 Karma
1 Solution

mchang_splunk
Splunk Employee
Splunk Employee

HEC will only extract timestamp from "Time" with epoch time as mentioned in the document:
https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_met...

For example, you must give the field name "time", otherwise, Splunk will use current time as timestamp:
curl -k "https://:8088/services/collector" -H "Authorization: Splunk " -d '{"host": "test", "index": "main", "sourcetype": "iks-log4j", "event": "2019-09-16T17:54:39,928 whatever.....", "time": 1568656479928}'

Or you can use raw event parsing:
https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Raw_event...

For example, the timestamp will be extracted from the time in "event":

curl https://:8088/services/collector/raw?channel= -H "Authorization: Splunk " -d '{"host": "test", "index": "main", "sourcetype": "iks-log4j", "data": {"event": "2019-09-16T17:54:39,928 whatever....."}'

So EITHER add "time" in event metadata OR use raw event parsing should be able to resolve this issue.

View solution in original post

0 Karma

mchang_splunk
Splunk Employee
Splunk Employee

HEC will only extract timestamp from "Time" with epoch time as mentioned in the document:
https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_met...

For example, you must give the field name "time", otherwise, Splunk will use current time as timestamp:
curl -k "https://:8088/services/collector" -H "Authorization: Splunk " -d '{"host": "test", "index": "main", "sourcetype": "iks-log4j", "event": "2019-09-16T17:54:39,928 whatever.....", "time": 1568656479928}'

Or you can use raw event parsing:
https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Raw_event...

For example, the timestamp will be extracted from the time in "event":

curl https://:8088/services/collector/raw?channel= -H "Authorization: Splunk " -d '{"host": "test", "index": "main", "sourcetype": "iks-log4j", "data": {"event": "2019-09-16T17:54:39,928 whatever....."}'

So EITHER add "time" in event metadata OR use raw event parsing should be able to resolve this issue.

View solution in original post

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!