You also need to tell the Heavy Forwarder to listen on port 9997 (or whatever you choose). They didn't include this step in the instructions to setup a heavy forwarder, but you can find it here:
So after spending ages (and a support call later) on installing a heavy forwarder, here the more detailed steps.
This is written up because most of the splunk documentation doesn't cover it or is flat out wrong.
This is to install a windows heavy forwarder to forward data to the splunk cloud.
1) Download splunk enterprise exe from the splunk site and install.
2) Log in and install your license (i had to contact support for this)
3) Remove the indexer roles.
Settings->health monitoring->Settings->General Setup, click on actions, un-tick search head and un-tick indexer. Save.
4) Download the SPL package from your splunk cloud (splunk calls this an "app" but it's just a bunch of settings). It is not the regular universal forwarder exe you get from splunk (do not install the separate universal forwarder software).
5) Run the following command on your Splunk Heavy Forwarder (or whatever path you install splunk too).
c:\program files\splunk\bin\splunk install app full_path_to_splunkclouduf.spl -auth username:password
6) Restart splunk
c:\program files\splunk\bin\splunk restart
7) Once splunk is restarted you'll need to check the correct outputs.conf is install
8) Make sure that C:\Program Files\Splunk\etc\apps\100_yourcloudname_splunkcloud\default\outsputs.conf is the same as C:\Program Files\Splunk\etc\system\local\outputs.conf
9) If the files above aren't the same, copy C:\Program Files\Splunk\etc\apps\100_yourcloudname_splunkcloud\default\outsputs.conf to C:\Program Files\Splunk\etc\system\local\outputs.conf and restart splunk.
10) Log in to your heavy forwarder and check the forwarders are now correct.
Settings->Forwarding and Receiving->Forward data
11) You can run this search on your splunk cloud to check if it's getting data from your forwarder.
index=_internal source=*metrics.log* group=tcpin_connections | stats values(version) by hostname fwdType os
I'm also having problems getting this to work. I followed the steps in this post. Can't find any straight answer from splunk docs, its a horrible mess.
My Splunk Cloud instance sees the Heavy Forwarder I setup, but its not receiving any logs.
On the Heavy Forwarder I get a ton of these entries:
03-13-2018 20:46:00.077 +0000 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 1700 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. 03-13-2018 20:46:10.090 +0000 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 1710 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
Steps are, essentially:
So it looks like doing this I am running into certificate problems. Splunkd.log doesn't show anything obvious, but the connection is timing out. I also had to make some changes to the outputs.conf file because it splunkd said one of the settings had a new name.
I am using windows for my heavy forwarder.
So I did this, and is says active forwards none, but shows my splunkcloud instances as configured but inactive.
I;m trying to forward Mcafee epo data that is being collected using the mcafee epo add-on and the Splunk DB connect.
I;m also seeing message in splunk saying:
skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block
12/1/2016, 5:14:54 PM
Forwarding to indexer group default-autolb-group blocked for 10 seconds.
12/1/2016, 5:13:22 PM
The search scheduler is disabled by the license Splunk is using. Scheduled searches that populate a summary index were found, but they will not be executed. This might affect dashboard panels that depend on the summary index. [!/help?location=learnmore.license.features Learn more]
12/1/2016, 5:12:50 PM
Also thanks a LOT! I really appreciate the help.