I'm trying to send data from Splunk universal forwarder (latest) to the Splunk cloud over HTTP event collector.
I have done the below steps:
1) Downloaded "Universal forwarder credentials" from our Splunk cloud and installed on Splunk universal forwarder machine.
2) Configured "outputs.conf" file as below
[httpout] |
Server.conf:
[proxyConfig] http_proxy =http://ip:port https_proxy = http://ip:port |
3) Tested using CURL command: I can send data to Splunk cloud
Response: {"text":"Success","code":0}
curl https://<splunk cloud endpoint:443> /services/collector -H "Authorization: Splunk <HEC TOKEN>" -d '{"event": "hello world"}' |
With the above configurations , I couldnot send data to Splunk cloud.. What do i miss here?
1) Where do I need to configure "inputs.conf" , "outputs.conf " and "server.conf" in ----> ...etc/system/local (OR) ...etc/apps/100_splunkcloud/local (OR) etc/apps/splunk_httpinput/local ?
2) If don't configure inputs.conf in local, as per the default inputs.conf, I should see _internal, _audit logs of UF right?
How can I troubleshoot this issue to send data from UF to Splunk cloud over http? Any help would be appreciated.
Thanks
MS
Splunkd.log:
08-21-2022 11:04:29.282 -0400 WARN TcpOutputFd [20567 TcpOutEloop] - Connect to <IP>9997 failed. Network is unreachable
08-21-2022 11:04:29.282 -0400 ERROR TcpOutputFd [20567 TcpOutEloop] - Connection to host=<IP>:9997 failed
To send data over HEC, Do I need to have 9997 port listening on the Splunk cloud servers?
Disable your tcpout output. A UF can send to either a tcpout or httpout. You can't have both.
@PickleRick
Do I need to disable tcpout in system/defaults as well?
See the https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/Wheretofindtheconfigurationfiles document. It will save you much pain later 😉
We are collecting logs from many source using HEC in Splunk cloud.
We have a requirement to collect data using universal forwarder. So, we are testing universal forwarder to send data to Splunk cloud over HEC. Also, I need to test if data is sent in compressed format.
If S2S works well for this scenario, Please provide me a guide on this.
Please see the "Send data to HTTP Event Collector on Splunk Cloud Platform" section in the following documentation.
https://docs.splunk.com/Documentation/Splunk/9.0.0/Data/UsetheHTTPEventCollector
Are you using the correct URI format with prefix and endpoint? The standard form for the HEC URI in Splunk Cloud Platform is as follows:
<protocol>://http-inputs-<host>.splunkcloud.com:<port>/<endpoint>
One thing to point out is that if you are using httpout, there is no need for the Splunk forwarder app (100_splunkcloud) as that is for Splunk-to-Spunk (S2S) forwarding. Splunk UFs can do either tcpout or httpout, but not both.
1. .conf files should generally be within a custom-created app for the purpose such as: $SPLUNK_HOME/etc/apps/network_inputs/local.conf or $SPLUNK_HOME/etc/apps/base_configs/server.conf, but can be created within $SPLUNK_HOME/etc/system/local without issue to have the highest global precedence if desired.
Here is some documentation on file precedence:
https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/Wheretofindtheconfigurationfiles
2. You are correct, _internal and _audit logs are collected by default in /etc/system/default/inputs.conf and will ingest into SplunkCloud once forwarding is configured.
I also came across this blog post which may be helpful:
https://discoveredintelligence.ca/solving-roaming-users-http-out-for-the-splunk-universal-forwarder/
I have configured below URI for [httpout] in outputs.conf file
uri = https://http-inputs-<host>.splunkcloud.com:443
Do I need to add <endpoint> as "/services/collector/_raw" in the uri?