Is it possible to know from which heavy forwarder syslog event got indexed by any specific fields in events

[Jayrsplunk]

Any fields in events or raw data holds information about HF through which it got indexed

[thambisetty]

If you are getting _internal logs from heavy forwarders where you are receiving syslog, then you can check based on source type to find out HF which is forwarding events to indexer by running below search

index=_internal metrics sourcetypename

[FrankVl]

How are you sending the syslog through those HFs? Plain UDP/TCP inputs in Splunk, or do you have a syslog daemon running that writes it to files, which are then picked up by the HF (this is the recommended approach by the way)?

If you have a syslog daemon writing to files on the HF, you can set that up in a way that it writes to a folder structure that includes the HF's hostname at some level. This way you can see from Splunk's source field which HF it came from.

Alternatively, you can configure each HF to write it's hostname to a newly added metadata field. But I've never fully understood whether that has any drawbacks, performance wise.

[Jayrsplunk]

Thanks FrankVI for the suggestions.

Yes. We have syslog daemon running that writes as files in HF.

[FrankVl]

In that case I would certainly look at adding the HF servername into the path of where the syslog daemon writes, such that that ends up in the source field.

You can also do this by creating a servername specific symlink to where syslog already writes the logs and then point splunk to monitor that location (use a wildcard to keep the splunk config simple and re-usable on each HF).

I've changed my comment to an answer, such that if it works out for you, you can accept it.

[jplumsdaine22]

Best practice is to use syslog-ng/rsyslog as a collector and then have either a universal forwarder collect the data or if you want to go state of the art use the HEC (https://conf.splunk.com/files/2017/slides/to-hec-with-syslog-scalable-aggregated-data-collection-in-...)

HFs have their place but you will definitely run into issues at scale. If you go down the HEC route (and given you already have an HF tier that should be easier to set up) it will be a doddle to add additional metadata.

For the best practice set up the best way to have forwarder attribution is to have syslog write to a path with some metadata in the path name. For example.... /var/log/$syslog-collector-name/$HOST/$FACILITY

Then the source field in splunk will contain the name of the syslog host the data was received at.

[coccyx]

This does not exist in Splunk today. Curious about why you need that information? That's something we could do with a product we're working on, but I'd like to understand your use case a bit better.

[Jayrsplunk]

Currently we are receiving syslog for about 5k devices/server which makes HF loaded so trying to add load balancing kind of mechanism so that load gets disributed across many HF's. In that case , if we have a field which says events comes from particular HF will be much helpful in case of any issues.

[jplumsdaine22]

If you're overwriting the host field with the value from the syslog message, then no (for an HF). splunk_server will contain the indexer name, but I'm assuming you have an HF collecting syslog and sending to them an indexing tier.

[Jayrsplunk]

yes..we are overwriting host value. Syslog collected in HF is send to indexer.