Heavy Forwarder System Logs - Best Practices

[rbakeredfi]

I have Heavy Forwarders that are running on Windows and Linux servers that still need to be monitored. Are there best practices for what to and not to log from a Heavy Forwarder?

For example, can I take my default Windows inputs.conf file from my Universal Forwarders and apply it to my Heavy Forwarders or will this cause a "logging loop" where the Heavy Forwarder is logging itself logging?

I am completely guessing but maybe I could copy over my UF inputs.conf file but disable the wineventlog:application logs? What would be the equivalent on a Linux HF?

[gcusello]

Hi @rbakeredfi ,

the first thing is to manage all the Forwarders (Universal and Heavy) using a Deployment Server, so you're sure to have the correct configurations on all machines.

Then I prefer to directly send UFs logs to the Indexers.

It's also possible to use HFs as concentrators if you want to reduce the connections between networs.

For syslogs, I hint to use two UFs or two HF as receivers, using a Load Balancer to balance traffic and manage fail over.

On these servers use rsyslog or syslog-ng to receive syslogs and then read these logs from the files.

About the logs from the HFs, you can install on these machines the appropriate add-ons and use them to monitor these machines.

Ciao.

Giuseppe

[rbakeredfi]

So it is possible to control with a deployment server? I thought I saw somewhere that it was not.

Which TAs are you referencing for the HFs? I am currently modifying a single Windows inputs.conf file and just pushing it to different machines via the deployment server. Which is where the root of the question, which inputs should definitely be turned off to avoid problems? 

[richgalloway]

HFs can log their OS/server just the same as a UF can.  Use the same TAs as on UFs - don't just copy inputs.conf files.

[rbakeredfi]

Which TAs are you referencing (for a Windows HF)? I have a Windows inputs.conf file that I'm sure came from an app, but I'm not sure which, that is being modified for certain needs now. If you had a specific TA in mind that may help determine which inputs are not suitable for a HF.

[richgalloway]

Putting inputs.conf on a HF without a matching props.conf means the events may not be indexed properly.  That's why I advise installing a TA.  Use the same TAs you use on the UFs.  If you don't have one, try Splunk Add-on for Microsoft Windows (https://splunkbase.splunk.com/app/742) and Splunk Add-on for Unix and Linux (https://splunkbase.splunk.com/app/833).

[rbakeredfi]

Taking "Splunk Add-on for Microsoft Windows (https://splunkbase.splunk.com/app/742)" for example, all of those inputs are disabled by default. The root of my question is which inputs should remain disabled to avoid causing issues with a Heavy Forwarder.

For example, if Splunk logging/forwarding logs were logged to the local wineventlogs:application, then would enabling wineventlogs:application cause an infinite loop of logging that it is logging?

[mmccul]

The question of which inputs to enable or not is always a "the inputs that provide the logs you care about".  You don't have to worry about the HF vs UF question in this area.

That said, make sure you aren't routing through a HF just for the sake of it.  HFs should only be typically used for one of a few reasons:

1. When you cannot route (actual routing, not just firewall) from the UF to the IDX (either on time schedule or permanently)
2. When you need modular inputs

Almost every other scenario, it's best to not go through a HF.  Your question makes me suspect you may be routing through a HF.

When a HF (or IDX) receives logs from another source, it will "cook" or parse the logs then send them according to the outputs.conf.  There are no log types being discussed here that put you in danger of a logarithmic volume growth or logging loop.   

[gcusello]

Hi @rbakeredfi ,

I usually enable wineventlog, and eventually some script.

Ciao.

Giuseppe