Getting Data In

How to set up a heavy forwarder to forward data to Splunk Cloud?

jgorman_THG
Explorer

HI!

I am setting-up a heavy forwarder to forward data to Splunk Cloud.

Do I just follow the instructions for setting-up a Universal forwarder to forward to Splunk Cloud? What address do I use as my recipient address?

Thanks,

JG

dwartaarcos
Engager

You also need to tell the Heavy Forwarder to listen on port 9997 (or whatever you choose). They didn't include this step in the instructions to setup a heavy forwarder, but you can find it here:

https://docs.splunk.com/Documentation/Splunk/7.3.1/Forwarding/Enableareceiver

ptsystemssuppor
Engager

So after spending ages (and a support call later) on installing a heavy forwarder, here the more detailed steps.
This is written up because most of the splunk documentation doesn't cover it or is flat out wrong.

This is to install a windows heavy forwarder to forward data to the splunk cloud.

1) Download splunk enterprise exe from the splunk site and install.
2) Log in and install your license (i had to contact support for this)
Settings->Licensing
3) Remove the indexer roles.
Settings->health monitoring->Settings->General Setup, click on actions, un-tick search head and un-tick indexer. Save.
4) Download the SPL package from your splunk cloud (splunk calls this an "app" but it's just a bunch of settings). It is not the regular universal forwarder exe you get from splunk (do not install the separate universal forwarder software).
https://yourcloudname.splunkcloud.com/en-US/app/splunkclouduf/setupuf
5) Run the following command on your Splunk Heavy Forwarder (or whatever path you install splunk too).

c:\program files\splunk\bin\splunk install app full_path_to_splunkclouduf.spl -auth username:password

6) Restart splunk
c:\program files\splunk\bin\splunk restart
7) Once splunk is restarted you'll need to check the correct outputs.conf is install
8) Make sure that C:\Program Files\Splunk\etc\apps\100_yourcloudname_splunkcloud\default\outsputs.conf is the same as C:\Program Files\Splunk\etc\system\local\outputs.conf
9) If the files above aren't the same, copy C:\Program Files\Splunk\etc\apps\100_yourcloudname_splunkcloud\default\outsputs.conf to C:\Program Files\Splunk\etc\system\local\outputs.conf and restart splunk.
10) Log in to your heavy forwarder and check the forwarders are now correct.
Settings->Forwarding and Receiving->Forward data
11) You can run this search on your splunk cloud to check if it's getting data from your forwarder.

index=_internal source=*metrics.log* group=tcpin_connections | stats values(version) by hostname fwdType os

vhebert17
New Member

I'm also having problems getting this to work. I followed the steps in this post. Can't find any straight answer from splunk docs, its a horrible mess.

My Splunk Cloud instance sees the Heavy Forwarder I setup, but its not receiving any logs.

On the Heavy Forwarder I get a ton of these entries:

03-13-2018 20:46:00.077 +0000 WARN  TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 1700 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
03-13-2018 20:46:10.090 +0000 WARN  TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 1710 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
0 Karma

sachaz
Explorer

Same here, how you resolve this?

0 Karma

saurabh_tek11
Communicator

Hi @jgorman_THG, There was congestion at the indexer side which was blocking the data indexing apparently because of Disk space.

0 Karma

pgreer_splunk
Splunk Employee
Splunk Employee

Steps are, essentially:

  1. Download the Heavy Forwarder
  2. Install, ./splunk enable boot-start
  3. Configure the HWF to receive data
  4. Configure the HWF to send data to indexer ( ./splunk add forward-server customer.splunkcloud.com:9997)
  5. Download your Universal Forwarder app from your Splunk Cloud instance
  6. Install the app at the CLI on your HWF (./splunk install app /tmp/100_customername_splunkcloud.spl -update 1 )
  7. Check HWF configs ( ./splunk list forward-server)
  8. List item

dionrivera
Path Finder

Thanks for the short but awesome list of steps. This helped me more than any other documentation (that I've discovered) in Splunk docs. I would add that if all you're doing is forwarding and not indexing on a host. You can skip step 3(receiving data)

0 Karma

dionrivera
Path Finder

Correction: After building a second HF. You do need to configure receiving. Receiving will allow the HF to receive data from forwarding clients.

0 Karma

ptsystemssuppor
Engager

Please don't follow these instructions, they are not complete.

0 Karma

kirknicholson
Engager

agreed, use brief instructions from pgreer_splunk

0 Karma

jgorman_THG
Explorer

Hi!

So it looks like doing this I am running into certificate problems. Splunkd.log doesn't show anything obvious, but the connection is timing out. I also had to make some changes to the outputs.conf file because it splunkd said one of the settings had a new name.

I am using windows for my heavy forwarder.

Any ideas?

Thanks,

JG

0 Karma

jgorman_THG
Explorer

HI!

So I did this, and is says active forwards none, but shows my splunkcloud instances as configured but inactive.

I;m trying to forward Mcafee epo data that is being collected using the mcafee epo add-on and the Splunk DB connect.

I;m also seeing message in splunk saying:

skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block
12/1/2016, 5:14:54 PM
Forwarding to indexer group default-autolb-group blocked for 10 seconds.
12/1/2016, 5:13:22 PM
The search scheduler is disabled by the license Splunk is using. Scheduled searches that populate a summary index were found, but they will not be executed. This might affect dashboard panels that depend on the summary index. [!/help?location=learnmore.license.features Learn more]
12/1/2016, 5:12:50 PM

Any ideas?

Also thanks a LOT! I really appreciate the help.

JG

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...