Working on better alerting on indexing volume/license usage and the like and I've stumbled across something in-explicable. We have 4 Heavy Forwarders that all have default/outputs.conf with [tcpout] indexAndForward = false
Now when I look at:
index=_internal host=<nameing convention of Splunk infrastructure devices) source="*metrics.log" group="per_index_thruput" series=main| timechart span=30m per_second(kb) BY host
One of the heavy forwarders shows up in the list of hosts (along with the indexers I'd expect)
Even more confusing is when I look at:
index=_internal source=*license_usage.log type="RolloverSummary"
This returns the vast majority of events from our license master with "pool" listed as the "auto_generated_pool_enterprise" as I'd expect. But it also shows all 4 of our heavy forwarders with "pool" listed as "auto_generated_pool_download-trial" or "auto_generated_pool_forwarder"
Looking at the Distributed Management Console app, under License Usage... if I look at all pools split by pool, I see all 3 of these pools (download-trial, enterprise and forwarder). When I look at it split by indexer, the list of indexers is in GUID so it's hard to correlate back to device, but there's 10 + "Other" listed... and we only have 5 indexers in our environment... so there's at least 5 more than expected.
I'd like to get this cleaned up so we can be certain that we're accurately reporting on which devices are consuming license and at what rate.
Anyone know why one HF would be found under
index=_internal source="*metrics.log" group="per_index_thruput" series=main
and all of them would be found under:
index=_internal source=*license_usage.log type="RolloverSummary"
As stated above... index and forward is false for all of them.
Thank you!
