Can forward logs to third party server from HF over HTTP

[sudha_krish]

I want to forward the logs to third party server from heavy forwarder over http.
Here is my outputs.conf

[httpout]
defaultGroup = otel_hec_group

[httpout:otel_hec_group]
#server = thirdparty_server:8443
uri = http://thirdparty_server:8443
useSSL = false
sourcetype = hf_to_otel
disabled = false
sslVerifyServerCert = false
headers = {"Host": "hf_server", "Content-Type": "application/json"}
timeout = 30


but i don't receive logs in third party server and i don't find any error in splunkd logs aswell.


@SplunkSE 

[sudha_krish]

@livehybrid  @PickleRick  @gcusello  Thanks for your responses. I found in the Splunk documentation that forwarding logs to third-party systems is typically done over TCP. I tried using TCP, but I did not receive Splunk metadata like host, sourcetype, source, and index on the third-party system.

[gcusello]

Hi @sudha_krish ,

it's normal: metadata are forwarded only to other Splunk instances!

using syslogs, you forward only raw events and you must recognize, in the third party system, metadata from the raw events (e.g. host is usually in the beginning of the even after the timestamp).

Ciao.

Giuseppe

[PickleRick]

That is correct. When Splunk sends data over "plain TCP" connection it just sends raw event (there is some degree of configurability but AFAIR it's limited to sending syslog priority header, maybe a timestamp).

If you wanted to send the event metadata (sourcetype/source/host) along with the event you'd have to rewrite the event's raw contents. But if you want to retain the event and index it locally along with sending it out you most probably want to index it in an unchanged form. And it's where it's getting complicated.

You'd have to use the CLONE_SOURCETYPE functionality to duplicate your event and split its processing path. Then send the original one to your indexer(s) and the cloned one you can modify and route to your TCP output.

[livehybrid]

Hi @sudha_krish 

httpout sends Splunk2Splunk (S2S) data but over HTTP (HEC) rather than typical S2S port 9997, is this what you are trying to achieve? 

It is intended that this is used when you are not able to send data to a remote Splunk instance using typical S2S. 

As @gcusello has said, if you want to send to a non-Splunk system you should look into using syslog output which will send the raw data rather than Splunk-parsed S2S data.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[PickleRick]

That's not exactly right.

The httpout uses the same port as "ordinary" HEC input and uses the same token-based authorization but the data is sent using a S2S-over-HTTP protocol. It's not the same as normal /event endpoint uses. So while you indeed can use it in situations when normal "unknown" protocol connectivity is disallowed so that you can leverage HTTP proxy support and such, it's in no way a standard HTTP POST-based data pushing method.

So the answer to @sudha_krish is no - you can't use httpout output to send data out to a non-Splunk HTTP server. BTW, there is no "headers" parameter for any Splunk outputs, let alone httpout one.

[sudha_krish]

Thanks for your answer,  I found in the Splunk documentation that forwarding logs to third-party systems is typically done over TCP. I tried using TCP, but I did not receive Splunk metadata like host, sourcetype, source, and index on the third-party system

[isoutamo]

That's true as you are sending over pure TCP which is not s2s. Those fields are part of s2s' metadata information. If you want send also those you must add those into your data stream's payload part. You can use props.conf and transforms.conf to modify that as needed.

But what you are actually trying to do and why and where you try to send that data? Maybe there are some other way to get events there? I see some OTEL name here....

[gcusello]

Hi @sudha_krish ,

I'm not sure that's possible to forward logs to a third party using http, the usual way is syslog as described at https://docs.splunk.com/Documentation/SplunkCloud/9.3.2411/Forwarding/Forwarddatatothird-partysystem...

Anyway, http requires to use a token, did you created a token in the receiver? did you enabled it? did you passed it ot your output'

Ciao.

Giuseppe