Getting Data In

timestamp recognization for HEC input

mchang_splunk
Splunk Employee
Splunk Employee

I'm trying to ingest HEC input into Splunk and set up correct props.conf as below but timestamp is not getting extracted correctly (mainly milliseconds are not matching with index times).

[iks-log4j]
TIME_PREFIX=^
TIME_FORMAT= %Y-%m-%dT%H:%M:%S,%3N
MAX_TIMESTAMP_LOOKAHEAD=23
SHOULD_LINEMERGE=false

LINE_BREAKER=([\r\n]+)

TRUNCATE=999999

Here is the sample event raw data:
2019-09-16T17:54:39,928 10647234 | INFO | XXXXXXXXXXXXXXXXXX

Tags (1)
0 Karma
1 Solution

mchang_splunk
Splunk Employee
Splunk Employee

HEC will only extract timestamp from "Time" with epoch time as mentioned in the document:
https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_met...

For example, you must give the field name "time", otherwise, Splunk will use current time as timestamp:
curl -k "https://:8088/services/collector" -H "Authorization: Splunk " -d '{"host": "test", "index": "main", "sourcetype": "iks-log4j", "event": "2019-09-16T17:54:39,928 whatever.....", "time": 1568656479928}'

Or you can use raw event parsing:
https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Raw_event...

For example, the timestamp will be extracted from the time in "event":

curl https://:8088/services/collector/raw?channel= -H "Authorization: Splunk " -d '{"host": "test", "index": "main", "sourcetype": "iks-log4j", "data": {"event": "2019-09-16T17:54:39,928 whatever....."}'

So EITHER add "time" in event metadata OR use raw event parsing should be able to resolve this issue.

View solution in original post

0 Karma

mchang_splunk
Splunk Employee
Splunk Employee

HEC will only extract timestamp from "Time" with epoch time as mentioned in the document:
https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_met...

For example, you must give the field name "time", otherwise, Splunk will use current time as timestamp:
curl -k "https://:8088/services/collector" -H "Authorization: Splunk " -d '{"host": "test", "index": "main", "sourcetype": "iks-log4j", "event": "2019-09-16T17:54:39,928 whatever.....", "time": 1568656479928}'

Or you can use raw event parsing:
https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Raw_event...

For example, the timestamp will be extracted from the time in "event":

curl https://:8088/services/collector/raw?channel= -H "Authorization: Splunk " -d '{"host": "test", "index": "main", "sourcetype": "iks-log4j", "data": {"event": "2019-09-16T17:54:39,928 whatever....."}'

So EITHER add "time" in event metadata OR use raw event parsing should be able to resolve this issue.

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...