Getting Data In

timestamp recognization for HEC input

mchang_splunk
Splunk Employee
Splunk Employee

I'm trying to ingest HEC input into Splunk and set up correct props.conf as below but timestamp is not getting extracted correctly (mainly milliseconds are not matching with index times).

[iks-log4j]
TIME_PREFIX=^
TIME_FORMAT= %Y-%m-%dT%H:%M:%S,%3N
MAX_TIMESTAMP_LOOKAHEAD=23
SHOULD_LINEMERGE=false

LINE_BREAKER=([\r\n]+)

TRUNCATE=999999

Here is the sample event raw data:
2019-09-16T17:54:39,928 10647234 | INFO | XXXXXXXXXXXXXXXXXX

Tags (1)
0 Karma
1 Solution

mchang_splunk
Splunk Employee
Splunk Employee

HEC will only extract timestamp from "Time" with epoch time as mentioned in the document:
https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_met...

For example, you must give the field name "time", otherwise, Splunk will use current time as timestamp:
curl -k "https://:8088/services/collector" -H "Authorization: Splunk " -d '{"host": "test", "index": "main", "sourcetype": "iks-log4j", "event": "2019-09-16T17:54:39,928 whatever.....", "time": 1568656479928}'

Or you can use raw event parsing:
https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Raw_event...

For example, the timestamp will be extracted from the time in "event":

curl https://:8088/services/collector/raw?channel= -H "Authorization: Splunk " -d '{"host": "test", "index": "main", "sourcetype": "iks-log4j", "data": {"event": "2019-09-16T17:54:39,928 whatever....."}'

So EITHER add "time" in event metadata OR use raw event parsing should be able to resolve this issue.

View solution in original post

0 Karma

mchang_splunk
Splunk Employee
Splunk Employee

HEC will only extract timestamp from "Time" with epoch time as mentioned in the document:
https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_met...

For example, you must give the field name "time", otherwise, Splunk will use current time as timestamp:
curl -k "https://:8088/services/collector" -H "Authorization: Splunk " -d '{"host": "test", "index": "main", "sourcetype": "iks-log4j", "event": "2019-09-16T17:54:39,928 whatever.....", "time": 1568656479928}'

Or you can use raw event parsing:
https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Raw_event...

For example, the timestamp will be extracted from the time in "event":

curl https://:8088/services/collector/raw?channel= -H "Authorization: Splunk " -d '{"host": "test", "index": "main", "sourcetype": "iks-log4j", "data": {"event": "2019-09-16T17:54:39,928 whatever....."}'

So EITHER add "time" in event metadata OR use raw event parsing should be able to resolve this issue.

0 Karma
Get Updates on the Splunk Community!

Security Highlights | November 2022 Newsletter

 November 2022 2022 Gartner Magic Quadrant for SIEM: Splunk Named a Leader for the 9th Year in a RowSplunk is ...

Platform Highlights | November 2022 Newsletter

 November 2022 Skill Up on Splunk with our New Builder Tech Talk SeriesCan you build it? Yes you can! *play ...

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...