Getting Data In

Unable to extract timestamp from incoming API POST

proylea
Contributor

I have an index cluster with load balancer
a curl sending a JSON event to HEC

curl http://indexers-amazonaws.com:8088/services/collector -H 'Authorization: Splunk ???' -d '{"sourcetype": "bma","event": {"timestamp": "Sun Aug 11 19:00:00 GMT+10:00 2019","Username": "joblogs", "requestID": "???", "access-level": "1", "authentication": "success"}}'

Props that appears to work when I do it manually through data input

[bma]
INDEXED_EXTRACTIONS = json
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
TIME_PREFIX = {"timestamp": "
category = Structured
disabled = false
pulldown_type = 1

I've tried numerous variation of props

What am I missing???

0 Karma
1 Solution

proylea
Contributor

I had tried adding time as suggested but got errors, didn't realise it needed to be epoch time.
For those who are interested this format works:
curl http://indexers-amazonaws.com:8088/services/collector/event -H 'Authorization: Splunk ???' -d '{"sourcetype": "bma","time": "1565561700","event": {"Username": "jobloggs", "tokenID": "???", "access-level": "1", "authentication": "success"}}'

View solution in original post

0 Karma

proylea
Contributor

I had tried adding time as suggested but got errors, didn't realise it needed to be epoch time.
For those who are interested this format works:
curl http://indexers-amazonaws.com:8088/services/collector/event -H 'Authorization: Splunk ???' -d '{"sourcetype": "bma","time": "1565561700","event": {"Username": "jobloggs", "tokenID": "???", "access-level": "1", "authentication": "success"}}'

0 Karma

woodcock
Esteemed Legend

When using the /collector/event endpoint, you need to supply your timestamp while formatting your event, along with sourcetype, source and host; if you want to extract the timestamp from your raw data then you need to use the /collector/raw HEC endpoint instead.

proylea
Contributor

Thank you sir, very good.
I figured after to much playing around with props that my problem was with the event.
I had tried adding time as suggested but got errors, didn't realise it needed to be epoch time.

For those who are interested this format works:
curl http://indexers-amazonaws.com:8088/services/collector/event -H 'Authorization: Splunk ???' -d '{"sourcetype": "bma","time": "1565561700","event": {"Username": "jobloggs", "tokenID": "???", "access-level": "1", "authentication": "success"}}'

And the raw option also does the trick but the final event is not as tidy

woodcock
Esteemed Legend

You should unaccept my answer and accept your because mine was not it. Feel free to UpVote, though!

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Have you tried adding TIME_FORMAT = %a %b %d %H:%M:%S %Z%:z %Y" to props.conf?

---
If this reply helps you, Karma would be appreciated.
0 Karma

proylea
Contributor

Thanks Rich, yes I had

0 Karma
Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Dynamic formatting from XML events

This challenge was first posted on Slack #puzzles channelFor a previous puzzle, I needed a set of fixed-length ...

Enter the Agentic Era with Splunk AI Assistant for SPL 1.4

  🚀 Your data just got a serious AI upgrade — are you ready? Say hello to the Agentic Era with the ...

Stronger Security with Federated Search for S3, GCP SQL & Australian Threat ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...