How to trace or tag events to know which specific Heavy Forwarder the events have passed through?

[kearaspoor]

We've got more than a dozen Heavy Forwarders (HF) that are behind a pair of load balancers that handle all our system log (syslog) traffic. Is there any way to trace back or tag the events to know which specific HF they were passed through?

[MuS]

Hi kearaspoor,

You can always use props.conf and transforms.conf on each of the HWF to add or change any data going through this HWF. Take a look at this answer https://answers.splunk.com/answers/40848/how-can-i-rewrite-add-info-from-metadata-to-the-contents-of... which provides a solution to add a host and source into the _raw data.

Be advised that this will increase license usage and also could produce other complications - this is only to show you that it is possible 😉

Hope this helps ...

cheers, MuS

[kearaspoor]

Thanks for the suggestion. We're currently bumping our license limit so thank you for the warning about that particular complication.

I'll also have to ponder this a bit because I want to keep the existing host/source information I just want to add the heavy forwarder as a troubleshooting/monitoring function. Not to mention the complications of managing separate props/transforms on each heavy forwarder.. I hadn't previously considered that added complexity. But you've given me a good place to start. Thank you!

[jrbanks6]

On prem or cloud? Source IP and port against your firewall logs if it is the latter. Are all the events indexed in single file?

[kearaspoor]

It's a range of devices including routers, switches, firewall, various security and networking appliances so they're ending up in a half-dozen or so individual indexes.