How to send data from Universal forwarder to Splunk cloud over HTTP (HEC)?

[mala_splunk_91]

I'm trying to send data from Splunk universal forwarder (latest) to the Splunk cloud over HTTP event collector.

I have done the below steps:

1) Downloaded "Universal forwarder credentials" from our Splunk cloud and installed on Splunk universal forwarder machine.

2) Configured "outputs.conf" file as below       

[httpout]
httpEventCollectorToken = <http_token>
uri = https://<splunkcloud_url>:443 

Server.conf:

[proxyConfig]
http_proxy =http://ip:port
https_proxy = http://ip:port

3) Tested using CURL command:  I can send data to Splunk cloud  
Response: {"text":"Success","code":0}

curl https://<splunk cloud endpoint:443> /services/collector  -H "Authorization: Splunk <HEC TOKEN>" -d '{"event": "hello world"}'

 With the above configurations , I couldnot send data to Splunk cloud.. What do i miss here?

 1) Where do I need to configure "inputs.conf" , "outputs.conf " and "server.conf"  in ----> ...etc/system/local  (OR) ...etc/apps/100_splunkcloud/local   (OR)  etc/apps/splunk_httpinput/local   ?

2) If don't configure inputs.conf in local, as per the default inputs.conf, I should see _internal, _audit logs of UF right?

How can I troubleshoot this issue to send data from UF to Splunk cloud over http? Any help would be appreciated.

Thanks

MS

[mala_splunk_91]

Splunkd.log:

08-21-2022 11:04:29.282 -0400 WARN TcpOutputFd [20567 TcpOutEloop] - Connect to <IP>9997 failed. Network is unreachable
08-21-2022 11:04:29.282 -0400 ERROR TcpOutputFd [20567 TcpOutEloop] - Connection to host=<IP>:9997 failed

To send data over HEC, Do I need to have 9997 port listening on the Splunk cloud servers?

[PickleRick]

Disable your tcpout output. A UF can send to either a tcpout or httpout. You can't have both.

[mala_splunk_91]

@PickleRick 

Do I need to disable tcpout in system/defaults as well?

[PickleRick]

See the https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/Wheretofindtheconfigurationfiles document. It will save you much pain later 😉

[isoutamo]

For curiosity, what it the issue which you try to solve by using HEC instead of normal S2S protocol between UFs and SplunkCloud?

[mala_splunk_91]

@isoutamo 

Also, We wanted to send data via Http and not TCP...

Splunk S2S helps on this?

[mala_splunk_91]

We are collecting logs from many source using HEC in Splunk cloud.
We have a requirement to collect data using universal forwarder. So, we are testing universal forwarder to send data to Splunk cloud over HEC. Also, I need to test if data is sent  in compressed format.

If S2S works well for this scenario, Please provide me a guide on this. 

[mroenicke]

Please see the "Send data to HTTP Event Collector on Splunk Cloud Platform" section in the following documentation.

https://docs.splunk.com/Documentation/Splunk/9.0.0/Data/UsetheHTTPEventCollector

Are you using the correct URI format with prefix and endpoint? The standard form for the HEC URI in Splunk Cloud Platform is as follows:

<protocol>://http-inputs-<host>.splunkcloud.com:<port>/<endpoint>

One thing to point out is that if you are using httpout, there is no need for the Splunk forwarder app (100_splunkcloud) as that is for Splunk-to-Spunk (S2S) forwarding. Splunk UFs can do either tcpout or httpout, but not both. 

https://docs.splunk.com/Documentation/Forwarder/latest/Forwarder/Configureforwardingwithoutputs.conf...

1. .conf files should generally be within a custom-created app for the purpose such as: $SPLUNK_HOME/etc/apps/network_inputs/local.conf or $SPLUNK_HOME/etc/apps/base_configs/server.conf, but can be created within $SPLUNK_HOME/etc/system/local without issue to have the highest global precedence if desired. 

Here is some documentation on file precedence:

https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/Wheretofindtheconfigurationfiles

2. You are correct, _internal and _audit logs are collected by default in /etc/system/default/inputs.conf and will ingest into SplunkCloud once forwarding is configured. 

I also came across this blog post which may be helpful:

https://discoveredintelligence.ca/solving-roaming-users-http-out-for-the-splunk-universal-forwarder/

[mala_splunk_91]

@mroenicke 

I have configured below URI for [httpout] in outputs.conf file

uri = https://http-inputs-<host>.splunkcloud.com:443

Do I need to add <endpoint> as "/services/collector/_raw" in the uri?