Getting Data In

Should I use a heavy forwarder or indexer for this scenario?

Exeterengineeri
Explorer

Greetings,

I'm trying to figure out if there is an advantage to having a heavy forwarder over just an indexer in the following scenario:

  • All of the infrastructure is virtual and is on the same hypervisor. Resources are dedicated.
  • Firewall logs are the primary reason for possibly using a heavy forwarder.
  • No pre or post processing of the logs is required--we just want them indexed.
  • Search factor and replication factor are both set to one. We don't need replicated data or indexes.

On one hand, I understand that if I use a heavy forwarder, I can span the output across multiple indexers. On the other hand, why not just make this machine an indexer, itself, and if I want it to be more of a dedicated resource, just don't let any other UFs or HFs know about it. Yet it would still be part of the cluster. What are there pros and cons of each approach? Thanks in advance.

1 Solution

skoelpin
SplunkTrust
SplunkTrust

Excellent question @Exeterengineering

The only disadvantage of installing a heavy forwarder vs. an indexer would be the inability to do distributed searches, so the question is, do you need to scale horizontally sometime down the road or do you believe you have enough beef to do the searching?

Another factor to consider is how much data do you index per day?

View solution in original post

0 Karma

sjohnson_splunk
Splunk Employee
Splunk Employee

What kind of firewall? Checkpoint, Palo Alto, Cisco ASA?

There is a specific app for Checkpoint and depending on the volume and number of devices you need to collect from you might need to scale up. If you are just going to forward syslog data from the device then I would recommend that you send that to a syslog server first and then monitor the log files with a universal forwarder.

0 Karma

MuS
SplunkTrust
SplunkTrust

Firewall logs are the primary reason for possibly using a heavy forwarder.

If those logs are coming in over syslog I suggest to read this excellent post http://www.georgestarcher.com/splunk-success-with-syslog/

cheers, MuS

0 Karma

mhassan_splunk
Splunk Employee
Splunk Employee

I wrote an extensive blog on syslog-ng with splunk. You may find it useful:
http://blogs.splunk.com/2016/05/05/high-performance-syslogging-for-splunk-using-syslog-ng-part-1/

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Excellent question @Exeterengineering

The only disadvantage of installing a heavy forwarder vs. an indexer would be the inability to do distributed searches, so the question is, do you need to scale horizontally sometime down the road or do you believe you have enough beef to do the searching?

Another factor to consider is how much data do you index per day?

0 Karma

Exeterengineeri
Explorer

We very well could have a need to scale beyond what we have now. How about something like a syslog-ng server with a universal forwarder spreading the output to the indexers?

0 Karma

skoelpin
SplunkTrust
SplunkTrust

This is a great idea. If you want the ability to "scale horizontally" then you need to add more indexers and set up UF's on your syslog server. Adding heavy forwarders would prevent you from scaling since they do not have the ability to distribute your searches to other instances.

Exeterengineeri
Explorer

Thank you, and after some more reflection, I agree. Syslog-ng it is.

skoelpin
SplunkTrust
SplunkTrust

Great to hear! If this answered your question, can you please accept the answer?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...