HEC token use for the source to Splunk

[uagraw01]

Hello Splunkers!!

I have ingested data into Splunk from the source system using the URI "https://localhost:8088/services/collector" along with the HEC token. However, the data is not being displayed in Splunk with the appropriate sourcetype parsing, which is affecting the timestamp settings for the events.

The sourcetype and timestamp are currently being displayed as below.

My actual props.conf setting as below :

[agv_voot]
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
KV_MODE = json
pulldown_type = 1
TIME_PREFIX = ^\@timestamp
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N
TIMESTAMP_FIELDS = @timestamp
TRANSFORMS-trim_timestamp = trim_long_timestamp

transforms.conf

[trim_long_timestamp]
REGEX = (\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3})\d+(-\d{2}:\d{2})
FORMAT = $1

Please help to fix the proper parsing with correct sourcetype and timestamp.

[PickleRick · Accepted solution]

https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTinput#services.2Fcollector.2Fevent

services/collector/event

Sends timestamped events to HTTP Event Collector using the Splunk platform JSON event protocol when auto_extract_timestamp is set to "true" in the /event URL.

• An example of a timestamp is: 2017-01-02 00:00:00.
• If there is a timestamp in the event's JSON envelope, Splunk honors that timestamp first.
• If there is no timestamp in the event's JSON envelope, the merging pipeline extracts the timestamp from the event.
• If "time=xxx" is used in the /event URL then auto_extract_timestamp is disabled.
• Splunk supports timestamps using the Epoch format.

In other words - unless you specify your URI as /services/collector/event?auto_extract_timestamp=true your timestamp will _not_ be extracted from the event itself (Splunk will not even bother looking for it - it will either get the data from the json envelope or will assume current timestamp if there is no timestamp in the envelope). And even if the auto_extract_timestamp parameter is set to true, in cases listed above extraction is not performed either.

See also https://www.aplura.com/assets/pdf/hec_pipelines.pdf