Splunk Dev

Splunk using HTTP Event Collector (HEC) with services/collector/raw endpoint to send whole datafile

RetailOperation
Explorer

Hello,

we want to send a whole file via the http event collector to the endpoint services/collector/raw as descirbed here:
http://dev.splunk.com/view/event-collector/SP-CAAAE8Y

"curl https://http-inputs-.splunkcloud.com/services/collector/raw -H "X-Splunk-Request-Channel: FE0ECFAD-13D5-401B-847D-77833BD77131" -H "Authorization: Splunk BD274822-96AA-4DA6-90EC-18940FB2414C" -d '' -v"

As we are on Splunk 7 we do not use the "X-Splunk-Request-Channel"

The data is recieved by an heavy forwarder and parsed to the indexer cluster.
The data is arriving at the assigned index but the problem is that splunk handles the whole file as one event.
I created a sourcetype on a test environment which breaks up the events in single events when i upload the file manually.
But when i send this file to our productive environment via the HEC with /raw endpoint the file is still indexed as one event.

I tried to configure the sourcetype on all indexers in the index cluster -> was nor working
I tried to configure the sourcetype also at the heavy forwarders -> not working

Any suggestions what to do here?

thanks in advance

Tags (1)
0 Karma

starcher
Influencer

The event endpoint expects properly formed JSON and things broken into events. It bypasses parsing queues. If you send to RAW you treat it like any file pickup ingestion and do your own line breaking etc as you would any text stream multi line event.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

How to Break a Kubernetes Pod on Purpose and Let the AI Troubleshooting Agent Find ...

Let's say I've just rolled out a Kubernetes Deployment with a typo in the image tag. Nothing explodes. The pod ...

Splunk Add-on for Microsoft Windows 11.0.3: Parse Once, Search Faster

The updated Splunk Add-on for Microsoft Windows helps customers search Windows events faster by reducing ...

What's New in Splunk Enterprise Security 8.8 and Beyond

Threats move at machine speed. Keeping up should not depend on having the biggest budget, the largest team, or ...