topic Re: Send data from Universal forwarder to Splunk cloud over HTTP (HEC) in Getting Data In

How to send data from Universal forwarder to Splunk cloud over HTTP (HEC)?

mala_splunk_91 — Mon, 22 Aug 2022 12:06:46 GMT

I'm trying to send data from Splunk universal forwarder (latest) to the Splunk cloud over HTTP event collector.

I have done the below steps:

1) Downloaded "Universal forwarder credentials" from our Splunk cloud and installed on Splunk universal forwarder machine.

2) Configured "outputs.conf" file as below

[httpout]
httpEventCollectorToken = <http_token>
uri = https://<splunkcloud_url>:443

Server.conf:

[proxyConfig]
http_proxy =http://ip:port
https_proxy = http://ip:port

3) Tested using CURL command: I can send data to Splunk cloud
Response: {"text":"Success","code":0}

curl https://<splunk cloud endpoint:443> /services/collector -H "Authorization: Splunk <HEC TOKEN>" -d '{"event": "hello world"}'

With the above configurations , I couldnot send data to Splunk cloud.. What do i miss here?

1) Where do I need to configure "inputs.conf" , "outputs.conf " and "server.conf" in ----> ...etc/system/local (OR) ...etc/apps/100_splunkcloud/local (OR) etc/apps/splunk_httpinput/local ?

2) If don't configure inputs.conf in local, as per the default inputs.conf, I should see _internal, _audit logs of UF right?

How can I troubleshoot this issue to send data from UF to Splunk cloud over http? Any help would be appreciated.

Thanks

Re: Send data from Universal forwarder to Splunk cloud over HTTP (HEC)

mala_splunk_91 — Sun, 21 Aug 2022 15:13:08 GMT

Splunkd.log:

08-21-2022 11:04:29.282 -0400 WARN TcpOutputFd [20567 TcpOutEloop] - Connect to <IP>9997 failed. Network is unreachable
08-21-2022 11:04:29.282 -0400 ERROR TcpOutputFd [20567 TcpOutEloop] - Connection to host=<IP>:9997 failed

To send data over HEC, Do I need to have 9997 port listening on the Splunk cloud servers?

Re: Send data from Universal forwarder to Splunk cloud over HTTP (HEC)

mroenicke — Sun, 21 Aug 2022 23:00:26 GMT

Please see the "Send data to HTTP Event Collector on Splunk Cloud Platform" section in the following documentation.

https://docs.splunk.com/Documentation/Splunk/9.0.0/Data/UsetheHTTPEventCollector

Are you using the correct URI format with prefix and endpoint? The standard form for the HEC URI in Splunk Cloud Platform is as follows:

<protocol>://http-inputs-<host>.splunkcloud.com:<port>/<endpoint>

One thing to point out is that if you are using httpout, there is no need for the Splunk forwarder app (100_splunkcloud) as that is for Splunk-to-Spunk (S2S) forwarding. Splunk UFs can do either tcpout or httpout, but not both.

https://docs.splunk.com/Documentation/Forwarder/latest/Forwarder/Configureforwardingwithoutputs.conf#Available_parameters_for_the_httpout_stanza

1. .conf files should generally be within a custom-created app for the purpose such as: $SPLUNK_HOME/etc/apps/network_inputs/local.conf or $SPLUNK_HOME/etc/apps/base_configs/server.conf, but can be created within $SPLUNK_HOME/etc/system/local without issue to have the highest global precedence if desired.

Here is some documentation on file precedence:

https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/Wheretofindtheconfigurationfiles

2. You are correct, _internal and _audit logs are collected by default in /etc/system/default/inputs.conf and will ingest into SplunkCloud once forwarding is configured.

I also came across this blog post which may be helpful:

https://discoveredintelligence.ca/solving-roaming-users-http-out-for-the-splunk-universal-forwarder/

Re: Send data from Universal forwarder to Splunk cloud over HTTP (HEC)

PickleRick — Mon, 22 Aug 2022 07:45:28 GMT

Disable your tcpout output. A UF can send to either a tcpout or httpout. You can't have both.

Re: Send data from Universal forwarder to Splunk cloud over HTTP (HEC)

isoutamo — Mon, 22 Aug 2022 09:46:35 GMT

For curiosity, what it the issue which you try to solve by using HEC instead of normal S2S protocol between UFs and SplunkCloud?

Re: Send data from Universal forwarder to Splunk cloud over HTTP (HEC)

mala_splunk_91 — Mon, 22 Aug 2022 12:48:36 GMT

@PickleRick

Do I need to disable tcpout in system/defaults as well?

Re: Send data from Universal forwarder to Splunk cloud over HTTP (HEC)

mala_splunk_91 — Mon, 22 Aug 2022 12:53:25 GMT

We are collecting logs from many source using HEC in Splunk cloud.
We have a requirement to collect data using universal forwarder. So, we are testing universal forwarder to send data to Splunk cloud over HEC. Also, I need to test if data is sent in compressed format.

If S2S works well for this scenario, Please provide me a guide on this.

Re: Send data from Universal forwarder to Splunk cloud over HTTP (HEC)

PickleRick — Mon, 22 Aug 2022 12:55:10 GMT

See the https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/Wheretofindtheconfigurationfiles document. It will save you much pain later 😉

Re: Send data from Universal forwarder to Splunk cloud over HTTP (HEC)

mala_splunk_91 — Mon, 22 Aug 2022 13:06:38 GMT

@mroenicke

I have configured below URI for [httpout] in outputs.conf file

uri = https://http-inputs-<host>.splunkcloud.com:443

Do I need to add <endpoint> as "/services/collector/_raw" in the uri?

Re: Send data from Universal forwarder to Splunk cloud over HTTP (HEC)

mala_splunk_91 — Mon, 22 Aug 2022 14:07:29 GMT

@isoutamo

Also, We wanted to send data via Http and not TCP...

Splunk S2S helps on this?