Getting Data In

How to reduce index size on a Heavy Forwarder

Contributor

We use a heavy forwarder to read and transmit data from a Windows Event Collectors "Forwarded Events".
The license is set to "Forwarder License".


The databases of the forwarder grew quite big and are almost filling up the disk space of the collector machine.

How do we reduce the index size of the forwarder?

Is it cached data ready to get sent OR data it has already sent that is stored in the local databases?

Tags (3)

SplunkTrust
SplunkTrust

Hi FRoth

open up the guide again and find this:

You can use Splunk Web to perform one other configuration (for heavy forwarders only). To store a copy of indexed data local to the forwarder:
1. From Forwarding and receiving, select Forwarding defaults.
2. Select Yes to store and maintain a local copy of the indexed data on the forwarder. 

just undo it or you set indexAndForward in outputs.conf to false, read more here

cheers,
MuS

Legend

But you chose "no" for this step in the instructions:

  1. Select Yes to store and maintain a local copy of the indexed data on the forwarder.

After you set all of the configurations in the heavy forwarder, did you restart it?

I suggest that you give the following commands on the heavy forwarder
1. splunk stop
2. splunk clean eventdata -index main
3. splunk start

If the index begins to grow again, then you have a configuration problem somewhere.

Contributor

"no" is already set.

I use the splunk heavy forwarder instance to send syslog to a syslog server on which runs splunk and indexes the data written by the syslog server.
(this is necessary because I use syslog-ng to filter the data AND provide access to the data for other tools. These tools run on the 20-30 GB full data set while splunk indexes only a 3GB subset)


I followed these instructions.

Could that be a cause for the indexing? Do I have to clear the index manually?

0 Karma

Contributor

That might be the case. 😉

I followed the description on this documentation page to deploy the heavy forwarder.

It says "Important: A heavy forwarder has a key advantage over light and universal forwarders in that it can index your data locally, as well as forward the data to another Splunk index. However, local indexing is turned off by default."


In my case it seems that indexing is turned on.

How do I turn it off?

0 Karma

Legend

This doesn't seem like a pure forwarder. To me it looks like you have an indexAndForward setup, so that it not just forwards the events it receives, but indexes them itself as well.

0 Karma

Contributor

Overview

0 Karma

Legend

Which indexes/databases are taking up the space?

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!