How to set up a heavy forwarder to forward data to Splunk Cloud?

[jgorman_THG]

HI!

I am setting-up a heavy forwarder to forward data to Splunk Cloud.

Do I just follow the instructions for setting-up a Universal forwarder to forward to Splunk Cloud? What address do I use as my recipient address?

Thanks,

JG

[dwartaarcos]

You also need to tell the Heavy Forwarder to listen on port 9997 (or whatever you choose). They didn't include this step in the instructions to setup a heavy forwarder, but you can find it here:

https://docs.splunk.com/Documentation/Splunk/7.3.1/Forwarding/Enableareceiver

[ptsystemssuppor]

So after spending ages (and a support call later) on installing a heavy forwarder, here the more detailed steps.
This is written up because most of the splunk documentation doesn't cover it or is flat out wrong.

This is to install a windows heavy forwarder to forward data to the splunk cloud.

1) Download splunk enterprise exe from the splunk site and install.
2) Log in and install your license (i had to contact support for this)
Settings->Licensing
3) Remove the indexer roles.
Settings->health monitoring->Settings->General Setup, click on actions, un-tick search head and un-tick indexer. Save.
4) Download the SPL package from your splunk cloud (splunk calls this an "app" but it's just a bunch of settings). It is not the regular universal forwarder exe you get from splunk (do not install the separate universal forwarder software).
https://yourcloudname.splunkcloud.com/en-US/app/splunkclouduf/setupuf
5) Run the following command on your Splunk Heavy Forwarder (or whatever path you install splunk too).

c:\program files\splunk\bin\splunk install app full_path_to_splunkclouduf.spl -auth username:password

6) Restart splunk
c:\program files\splunk\bin\splunk restart
7) Once splunk is restarted you'll need to check the correct outputs.conf is install
8) Make sure that C:\Program Files\Splunk\etc\apps\100_yourcloudname_splunkcloud\default\outsputs.conf is the same as C:\Program Files\Splunk\etc\system\local\outputs.conf
9) If the files above aren't the same, copy C:\Program Files\Splunk\etc\apps\100_yourcloudname_splunkcloud\default\outsputs.conf to C:\Program Files\Splunk\etc\system\local\outputs.conf and restart splunk.
10) Log in to your heavy forwarder and check the forwarders are now correct.
Settings->Forwarding and Receiving->Forward data
11) You can run this search on your splunk cloud to check if it's getting data from your forwarder.

index=_internal source=*metrics.log* group=tcpin_connections | stats values(version) by hostname fwdType os

[vhebert17]

I'm also having problems getting this to work. I followed the steps in this post. Can't find any straight answer from splunk docs, its a horrible mess.

My Splunk Cloud instance sees the Heavy Forwarder I setup, but its not receiving any logs.

On the Heavy Forwarder I get a ton of these entries:

03-13-2018 20:46:00.077 +0000 WARN  TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 1700 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
03-13-2018 20:46:10.090 +0000 WARN  TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 1710 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

[sachaz]

Same here, how you resolve this?

[saurabh_tek11]

Hi @jgorman_THG, There was congestion at the indexer side which was blocking the data indexing apparently because of Disk space.

[pgreer_splunk]

Steps are, essentially:

1. Download the Heavy Forwarder
2. Install, ./splunk enable boot-start
3. Configure the HWF to receive data
4. Configure the HWF to send data to indexer ( ./splunk add forward-server customer.splunkcloud.com:9997)
5. Download your Universal Forwarder app from your Splunk Cloud instance
6. Install the app at the CLI on your HWF (./splunk install app /tmp/100_customername_splunkcloud.spl -update 1 )
7. Check HWF configs ( ./splunk list forward-server)
8. List item

[dionrivera]

Thanks for the short but awesome list of steps. This helped me more than any other documentation (that I've discovered) in Splunk docs. I would add that if all you're doing is forwarding and not indexing on a host. You can skip step 3(receiving data)

[dionrivera]

Correction: After building a second HF. You do need to configure receiving. Receiving will allow the HF to receive data from forwarding clients.

[ptsystemssuppor]

Please don't follow these instructions, they are not complete.

[kirknicholson]

agreed, use brief instructions from pgreer_splunk

[jgorman_THG]

Hi!

So it looks like doing this I am running into certificate problems. Splunkd.log doesn't show anything obvious, but the connection is timing out. I also had to make some changes to the outputs.conf file because it splunkd said one of the settings had a new name.

I am using windows for my heavy forwarder.

Any ideas?

Thanks,

JG

[jgorman_THG]

HI!

So I did this, and is says active forwards none, but shows my splunkcloud instances as configured but inactive.

I;m trying to forward Mcafee epo data that is being collected using the mcafee epo add-on and the Splunk DB connect.

I;m also seeing message in splunk saying:

skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block
12/1/2016, 5:14:54 PM
Forwarding to indexer group default-autolb-group blocked for 10 seconds.
12/1/2016, 5:13:22 PM
The search scheduler is disabled by the license Splunk is using. Scheduled searches that populate a summary index were found, but they will not be executed. This might affect dashboard panels that depend on the summary index. [!/help?location=learnmore.license.features Learn more]
12/1/2016, 5:12:50 PM

Any ideas?

Also thanks a LOT! I really appreciate the help.

JG