topic Re: Unable to extract timestamp from incoming API POST in Getting Data In

Unable to extract timestamp from incoming API POST

proylea — Sun, 11 Aug 2019 10:57:29 GMT

I have an index cluster with load balancer
a curl sending a JSON event to HEC

curl http://indexers-amazonaws.com:8088/services/collector -H 'Authorization: Splunk ???' -d '{"sourcetype": "bma","event": {"timestamp": "Sun Aug 11 19:00:00 GMT+10:00 2019","Username": "joblogs", "requestID": "???", "access-level": "1", "authentication": "success"}}'

Props that appears to work when I do it manually through data input

[bma]
INDEXED_EXTRACTIONS = json
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
TIME_PREFIX = {"timestamp": "
category = Structured
disabled = false
pulldown_type = 1

I've tried numerous variation of props

What am I missing???

Re: Unable to extract timestamp from incoming API POST

richgalloway — Sun, 11 Aug 2019 13:21:33 GMT

Have you tried adding TIME_FORMAT = %a %b %d %H:%M:%S %Z%:z %Y" to props.conf?

Re: Unable to extract timestamp from incoming API POST

woodcock — Sun, 11 Aug 2019 21:53:16 GMT

When using the /collector/event endpoint, you need to supply your timestamp while formatting your event, along with sourcetype, source and host; if you want to extract the timestamp from your raw data then you need to use the /collector/raw HEC endpoint instead.

Re: Unable to extract timestamp from incoming API POST

proylea — Sun, 11 Aug 2019 22:30:14 GMT

Thank you sir, very good.
I figured after to much playing around with props that my problem was with the event.
I had tried adding time as suggested but got errors, didn't realise it needed to be epoch time.

For those who are interested this format works:
curl http://indexers-amazonaws.com:8088/services/collector/event -H 'Authorization: Splunk ???' -d '{"sourcetype": "bma","time": "1565561700","event": {"Username": "jobloggs", "tokenID": "???", "access-level": "1", "authentication": "success"}}'

And the raw option also does the trick but the final event is not as tidy

Re: Unable to extract timestamp from incoming API POST

proylea — Sun, 11 Aug 2019 22:31:48 GMT

Thanks Rich, yes I had

Re: Unable to extract timestamp from incoming API POST

woodcock — Sun, 11 Aug 2019 22:45:03 GMT

You should unaccept my answer and accept your because mine was not it. Feel free to UpVote, though!

Re: Unable to extract timestamp from incoming API POST

proylea — Mon, 12 Aug 2019 11:32:46 GMT

I had tried adding time as suggested but got errors, didn't realise it needed to be epoch time.
For those who are interested this format works:
curl http://indexers-amazonaws.com:8088/services/collector/event -H 'Authorization: Splunk ???' -d '{"sourcetype": "bma","time": "1565561700","event": {"Username": "jobloggs", "tokenID": "???", "access-level": "1", "authentication": "success"}}'