Is there a way to search a log and figure out which heavy forwarder sent the log to the indexer?
Had a need for this today. This search does the trick for me:
index=_internal (host=myheavyforwarder1 OR host=myheavyforwarder2 OR host=myheavyforwarder3) sourcetype=splunkd "group=tcpin_connections" component=Metrics | timechart span=30m dc(hostname) by host
From this, I can see the distinct count of hostnames flowing through each heavy forwarder...
Btw, if you were to dig into these metrics, you could infer other things... like which logs went through which heavy forwarder by looking at group=per_source_thruput series="your source"
. The key is "infer" because your universal forwarders should load balance through your heavy forwarders and most logs will go through all of them.
With the combination of the per source thruput and a rough timeframe though, you could take a good guess.
I would say it depends on how you've setup log monitoring.
a)If you're not overriding the host, the host field in your log will denote the forwarder which is collected it.
your base search | stats count by host | table host
b) If you're overriding the host, then find the host (overridden) value in your logs and search for corresponding logging server in metrics logs
index=_internal sourcetype=splunkd source=*metrics.log group=per-host_thruput series="hostValueFromYourLog" | stats count by host | table host