Getting Data In

How to send data from Universal forwarder to Splunk cloud over HTTP (HEC)?

mala_splunk_91
Explorer

I'm trying to send data from Splunk universal forwarder (latest) to the Splunk cloud over HTTP event collector.

I have done the below steps:

1) Downloaded "Universal forwarder credentials" from our Splunk cloud and installed on Splunk universal forwarder machine.

2) Configured "outputs.conf" file as below       

[httpout]
httpEventCollectorToken = <http_token>
uri = https://<splunkcloud_url>:443 

 

Server.conf:

[proxyConfig]
http_proxy =http://ip:port
https_proxy = http://ip:port

 

3) Tested using CURL command:  I can send data to Splunk cloud  
Response: {"text":"Success","code":0}

curl https://<splunk cloud endpoint:443> /services/collector  -H "Authorization: Splunk <HEC TOKEN>" -d '{"event": "hello world"}'

 With the above configurations , I couldnot send data to Splunk cloud.. What do i miss here?

 1) Where do I need to configure "inputs.conf" , "outputs.conf " and "server.conf"  in ----> ...etc/system/local  (OR) ...etc/apps/100_splunkcloud/local   (OR)  etc/apps/splunk_httpinput/local   ?

2) If don't configure inputs.conf in local, as per the default inputs.conf, I should see _internal, _audit logs of UF right?

How can I troubleshoot this issue to send data from UF to Splunk cloud over http? Any help would be appreciated.

Thanks

MS

Tags (1)
0 Karma

mala_splunk_91
Explorer

Splunkd.log:

08-21-2022 11:04:29.282 -0400 WARN TcpOutputFd [20567 TcpOutEloop] - Connect to <IP>9997 failed. Network is unreachable
08-21-2022 11:04:29.282 -0400 ERROR TcpOutputFd [20567 TcpOutEloop] - Connection to host=<IP>:9997 failed

To send data over HEC, Do I need to have 9997 port listening on the Splunk cloud servers?

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Disable your tcpout output. A UF can send to either a tcpout or httpout. You can't have both.

0 Karma

mala_splunk_91
Explorer

@PickleRick 

Do I need to disable tcpout in system/defaults as well?

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust
0 Karma

isoutamo
SplunkTrust
SplunkTrust
For curiosity, what it the issue which you try to solve by using HEC instead of normal S2S protocol between UFs and SplunkCloud?
0 Karma

mala_splunk_91
Explorer

@isoutamo 

Also, We wanted to send data via Http and not TCP...

Splunk S2S helps on this?

 

0 Karma

mala_splunk_91
Explorer

We are collecting logs from many source using HEC in Splunk cloud.
We have a requirement to collect data using universal forwarder. So, we are testing universal forwarder to send data to Splunk cloud over HEC. Also, I need to test if data is sent  in compressed format.

If S2S works well for this scenario, Please provide me a guide on this. 

0 Karma

mroenicke
New Member

Please see the "Send data to HTTP Event Collector on Splunk Cloud Platform" section in the following documentation.

https://docs.splunk.com/Documentation/Splunk/9.0.0/Data/UsetheHTTPEventCollector

 

Are you using the correct URI format with prefix and endpoint? The standard form for the HEC URI in Splunk Cloud Platform is as follows:

<protocol>://http-inputs-<host>.splunkcloud.com:<port>/<endpoint>

 

One thing to point out is that if you are using httpout, there is no need for the Splunk forwarder app (100_splunkcloud) as that is for Splunk-to-Spunk (S2S) forwarding. Splunk UFs can do either tcpout or httpout, but not both. 

https://docs.splunk.com/Documentation/Forwarder/latest/Forwarder/Configureforwardingwithoutputs.conf...

 

1. .conf files should generally be within a custom-created app for the purpose such as: $SPLUNK_HOME/etc/apps/network_inputs/local.conf or $SPLUNK_HOME/etc/apps/base_configs/server.conf, but can be created within $SPLUNK_HOME/etc/system/local without issue to have the highest global precedence if desired. 

Here is some documentation on file precedence:

https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/Wheretofindtheconfigurationfiles

 

2. You are correct, _internal and _audit logs are collected by default in /etc/system/default/inputs.conf and will ingest into SplunkCloud once forwarding is configured. 

 

I also came across this blog post which may be helpful:

https://discoveredintelligence.ca/solving-roaming-users-http-out-for-the-splunk-universal-forwarder/

 

 

0 Karma

mala_splunk_91
Explorer

@mroenicke 

I have configured below URI for [httpout] in outputs.conf file

uri = https://http-inputs-<host>.splunkcloud.com:443

Do I need to add <endpoint> as "/services/collector/_raw" in the uri?

 

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...