Getting Data In

Unable to extract timestamp from incoming API POST

proylea
Contributor

I have an index cluster with load balancer
a curl sending a JSON event to HEC

curl http://indexers-amazonaws.com:8088/services/collector -H 'Authorization: Splunk ???' -d '{"sourcetype": "bma","event": {"timestamp": "Sun Aug 11 19:00:00 GMT+10:00 2019","Username": "joblogs", "requestID": "???", "access-level": "1", "authentication": "success"}}'

Props that appears to work when I do it manually through data input

[bma]
INDEXED_EXTRACTIONS = json
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
TIME_PREFIX = {"timestamp": "
category = Structured
disabled = false
pulldown_type = 1

I've tried numerous variation of props

What am I missing???

0 Karma
1 Solution

proylea
Contributor

I had tried adding time as suggested but got errors, didn't realise it needed to be epoch time.
For those who are interested this format works:
curl http://indexers-amazonaws.com:8088/services/collector/event -H 'Authorization: Splunk ???' -d '{"sourcetype": "bma","time": "1565561700","event": {"Username": "jobloggs", "tokenID": "???", "access-level": "1", "authentication": "success"}}'

View solution in original post

0 Karma

proylea
Contributor

I had tried adding time as suggested but got errors, didn't realise it needed to be epoch time.
For those who are interested this format works:
curl http://indexers-amazonaws.com:8088/services/collector/event -H 'Authorization: Splunk ???' -d '{"sourcetype": "bma","time": "1565561700","event": {"Username": "jobloggs", "tokenID": "???", "access-level": "1", "authentication": "success"}}'

0 Karma

woodcock
Esteemed Legend

When using the /collector/event endpoint, you need to supply your timestamp while formatting your event, along with sourcetype, source and host; if you want to extract the timestamp from your raw data then you need to use the /collector/raw HEC endpoint instead.

proylea
Contributor

Thank you sir, very good.
I figured after to much playing around with props that my problem was with the event.
I had tried adding time as suggested but got errors, didn't realise it needed to be epoch time.

For those who are interested this format works:
curl http://indexers-amazonaws.com:8088/services/collector/event -H 'Authorization: Splunk ???' -d '{"sourcetype": "bma","time": "1565561700","event": {"Username": "jobloggs", "tokenID": "???", "access-level": "1", "authentication": "success"}}'

And the raw option also does the trick but the final event is not as tidy

woodcock
Esteemed Legend

You should unaccept my answer and accept your because mine was not it. Feel free to UpVote, though!

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Have you tried adding TIME_FORMAT = %a %b %d %H:%M:%S %Z%:z %Y" to props.conf?

---
If this reply helps you, Karma would be appreciated.
0 Karma

proylea
Contributor

Thanks Rich, yes I had

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Quantify Your Splunk Investment Impact: Introducing Savings Metrics to Value Insights

Building on the foundation established in our initial Value Insights releases, we are introducing the Savings ...

Event Series: Telemetry Pipeline Management

Balancing Scale and Spend: Gaining Control Over High-Volume Metrics in Splunk Observability Cloud As ...

Kick the Tires Before You Commit: A Hands-On Tour of the Splunk Observability Cloud ...

Evaluating an enterprise observability platform usually goes like this: fill out a form, get a free trial with ...