timestamp extraction not working

[davidbann]

I have an http event collector configured with a heavy forwarder in the DMZ forwarding to an internal Indexer. The timtestamp of all events is being set to the time received, it's not picking up the "time" value from the body despite my props.conf settings. No errors or warnings in "_internal" around timestamp or anything close to it.

Test event sent to the collector:

curl --location --request POST 'https://<redacted>.com/services/collector' \
--header 'Authorization: Splunk <redacted>' \
--header 'Content-Type: application/json' \
--data-raw '{"event": {"time":"2021-02-04 20:20:20.123-05:00","userSettings":{"userId":"ab12345","userName":"ab12345,"site":"000"},"version":5070004},"sourcetype": "st-test"}'

shows up as expected in Search results as expected (raw):

{"time":"2021-02-04 20:20:20.123-05:00","userSettings":{"userId":"ab12345","userName":"ab12345","site":"901"},"version":5070004}

props.conf for this sourcetype is configured on both the heavy forwarder and internal indexer:

[st-test]
TRUNCATE = 100000
INDEXED_EXTRACTIONS = json
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Structured
disabled = false
pulldown_type = 1
TIME_PREFIX = "time":"
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3N%:z
MAX_TIMESTAMP_LOOKAHEAD = 32

Any ideas?

[bowesmana · Accepted solution]

@davidbann 

I believe you're possible using the wrong endpoint to support event timestamp extraction.  See

https://docs.splunk.com/Documentation/Splunk/8.1.2/RESTREF/RESTinput#services.2Fcollector.2Fevent

timestamp extraction is a bit finicky with HEC, but there is a short discussion of timestamp extraction there. There is an envelope timestamp and event timestamp and I recall when using this some months back, that you need to use the raw collector endpoint to get timestamp extracted using settings from props.conf. 

I forget all the detail, In our case we needed to resort to using auto_extract_timestamp=true in the /event endpoint. Have a play with this and the /raw endpoints.

[davidbann]

yep that did it. using:

/services/collector/event?auto_extract_timestamp=true

resulted in the timestamp being picked out of the body instead of event time.

Re-reading https://docs.splunk.com/Documentation/Splunk/8.1.1/Data/FormateventsforHTTPEventCollector

I noticed the following: "The HTTP Event Collector endpoint extracts the events from the HTTP request and parses them before sending them to indexers."

I think that explains the "finicky" behavior as it doesn;t follow the same path as other inputs.

Thanks @bowesmana !

[richgalloway]

When you look at the raw event in Search, what sourcetype is shown?

[davidbann]

st-test as expected