I am working on a matrix of data sources for my splunk deployment.
I need to map my data sources -collection method (for example firewalls, DC(s), servers and other appliances) from forwarders or syslog or other log feeds.
Many of the data sources were setup prior to my involvement.
How would one take on the is challenge and no - forwarder monitoring is not enabled.
I suggest a couple of things. First, just get a list of basic info in your Splunk instance by running this search command
| tstats count where index=* by host index sourcetype source
Of course, the host that appears in the list might be overridden at some point during parsing, but it is often the same host name as the original forwarder or source. It's a good starting point.
Second, the Splunk indexers will keep some information in their internal logs, which are automatically indexed into _internal. Splunk forwarders send their internal logs to the indexers by default. So even if you don't have forwarder monitoring, you should still have the fowarder logs unless this has been explicitly disabled by your predecessor. And you should turn on forwarder monitoring, it doesn't cost anything to do it.
You can also run a variety of searches against the internal index. The Metrics component only collects information about the top 10 sources during each time interval, so it will not be complete information, but this search may still be helpful:
index=_internal sourcetype=splunkd component=Metrics group=per_source_thruput | chart sum(ev) by series host | rename series as source
The above search shows the number of events for each source. If you don't care about that, you can get a more complete list from this search:
index=_internal sourcetype=splunkd component=WatchedFile | stats list(file) as Files by host
This shows all the file inputs. For searches in the _internal index, the host name is the name of the forwarder that is sending the data. In other indexes, the host name for the file will be based on the setting in inputs.conf, so these host names could be different.
And there is more, but this should get you started.
Not sure if you are still looking at this... but in my splunk DMC instance, under settings>monitor console> Forwarders: Instance, I can see the KB/s data rate per universal forwarder, however I cannot see a GB/day volume anywhere.
Do you have some search syntax to convert KB/S to GB/day volume? Or is there another report somewhere?