Note about how HEC events are parsed based on the Endpoint configured

[splunkreal]

Hello, a quick note from support to explain how events are handled on HEC (on HF for instance) :

https://help.splunk.com/en/splunk-enterprise/get-started/get-data-in/9.3/get-data-with-http-event-co...

Event endpoint:

1. The event is pre-structured JSON — event boundaries are defined by the JSON structure itself

1. LINE_BREAKER and EVENT_BREAKER settings in props.conf do NOT apply because the JSON payload already defines what constitutes an event

1. However, other props. conf/transforms.conf rules from a TA will still apply if the sourcetype matches — specifically index-time transforms (field extractions, routing, etc.)

1. The sender controls metadata (host, source, sourcetype, index, time) directly in the JSON payload, and individual events in a batch can override token defaults independently

Raw endpoint:

1. Data is treated as raw, unstructured text — just like file-based inputs

1. LINE_BREAKER (regex-based) and EVENT_BREAKER (intelligent breaking) rules from props.conf fully apply for the assigned sourcetype, matching the behaviour of file-monitored inputs

1. The sourcetype is determined by the HEC token configuration, not the payload

1. If a TA installed on the HF has props.conf rules matching that sourcetype (e.g., SHOULD_LINEMERGE, TIME_FORMAT, INDEXED_EXTRACTIONS), those rules will run at index time

1. For batch raw JSON events, props.conf must be configured correctly — e.g., INDEXED_EXTRACTIONS = json for JSON arrays, or SHOULD_LINEMERGE = false for line-delimited events

A TA typically ships two types of config:

props.conf / transforms.conf (index-time rules) — These apply on the HF only if the sourcetype of the incoming HEC data matches stanzas in the TA. With /raw, the full parsing pipeline runs (line breaking, timestamp parsing, field transforms). With /event, only the transforms/field extraction rules apply — not line breaking.

[jasonmiller11tt]

/event uses pre-structured JSON, so line breaking is skipped. /raw follows the normal parsing pipeline, meaning props.conf settings like LINE_BREAKER and INDEXED_EXTRACTIONS apply. The key is ensuring the incoming sourcetype matches your TA configuration. Thanks for the clear explanation!

[PickleRick]

That's also not entirely true. When you say "expects already-defined json events" one could interpret it as "you can only ingest events which are json structures using this endpoint" which is completely untrue. You can ingest any data you want, you just have to use the enpoint by means of sending appropriately formated json event containing the raw event as "event" field. See https://help.splunk.com/en/data-management/get-data-in/get-data-into-splunk-enterprise/10.4/get-data... for details.

[henry_collins]

I think that's a really helpful summary. The biggest takeaway is that /event expects already-defined JSON events, so event breaking settings aren't used, while /raw goes through the normal parsing pipeline where props.conf settings like LINE_BREAKER and SHOULD_LINEMERGE matter. Keeping that distinction in mind makes it much easier to choose the right endpoint for your data.

[PickleRick]

Yup, but the rules regarding HEC are not Kafka-addon-specific 🙂

[PickleRick]

This is not 100% true. Most is but not all.

With /event endpoint:

1. The source _can_ provide index/source/sourcetype/host values but if it doesn't the input assumes the default values. Also if a source requests that event be sent to an index which is not allowed with the provided HEC token, the request will fail
2. Unless explicitly requested with a proper URI parameter, timestamp recognition does _not_ happen with /event endpoint. And even if you want Splunk to parse timestamp but provide one on your own, time parsing won't happen.

For /raw endpoint:

1. To be fully honest, I don't think I've ever tested it with a not-line-oriented sourcetype but I wouldn't be surprised if it did breaking on a per-line basis. But that's just my hunch. Might as well be as you're saying so just test it for yourself if you wanna rely on it. Most notably I'm not sure how the input will react to different parts of a multiline events sent with different HTTP requests. I would be very surprised if it had a way to "correlate" those two separate requests into a single line. So test before use!
2. You _can_ provide destination metadata with your request by means of URI parameters or custom headers. But you have to do it for a whole HTTP request, not for each single event separately.

[splunkreal]

Hi @PickleRick 

We use /event endpoint, our Kafka expert told me he extracts the “message” field and sends it to our HEC through the connector. Does this mean HEC receives text or json events?

He also told me messages are delimited in json, providing source and sourcetype for instance however I understand we don’t control timestamp from Splunk’s side. It’s better than raw endpoint as otherwise he would need to add delimiter on the Vector’s agent source or hardcoded for network logs for example.

Thanks.

[PickleRick]

Well, I can't know how your Kafka pipeline is configured.

But the thing is that when you send events to the HEC endpoint they look like this (quote from the docs; a syntactically erroneous one since json doesn't allow comments 😁)

{
    "time": 1426279439, // epoch time
    "host": "localhost",
    "source": "random-data-generator",
    "sourcetype": "my_sample_data",
    "index": "main",
    "event":  "Hello world!" 
}

 All fields except "event" are optional - if there is no time supplied, Splunk will assume current time (or if sent with ?auto_extract_timestamp=true it wil be parsed from the event); for the rest of the fields Splunk will assume the defaults defined for particular HEC token. You can also send additional indexed fields as a "fields" structure.

Judging from your descripiton I'd assume that whatever the "message" field on your Kafka side is, it's getting pushed as "event" field into the HEC input. But what is in your "message" field in Kafka - that I don't know.

And if you use /services/collector/event?auto_extract_timestamp=true and _not_ provide the "time" field, Splunk will parse the timestamp out of the event regarding to the normal settings for the sourcetype. Of course if you have a reliable timestamp at the Kafka level already, you can just as well send it as "time" field and save Splunk some time parsing it again 🙂

[cartergray70543]

Good points. I'd especially recommend testing /raw with multiline events before relying on it, as behavior across separate requests can be tricky. The note about /event only parsing timestamps when explicitly requested is also an important distinction.

[splunkreal]

Thanks @PickleRick  for your reply, we use Kafka Connect for your information with event endpoint. The most important is HF addons in most case will apply parsing, fields extractions... however looks like timestamp will be controlled by Kafka.