Breakdown by Sourcetype from Each Forwarder

[reed_kelly]

We have a number of heavy forwarders sending cooked data to our indexers. We can get the total KBs sent by each forwarder by searching the metrics.log files. Is there a way to get a breakdown by sourcetype of data sent by each forwarder? Maybe this is a search that we can run from each forwarder?

[rgcurry]

In the Search app, the third panel down, the panel on the left displays all the sourcetypes and some data about them. Is this what you seek?

[rgcurry]

Does this give you what you are looking for?

index=* sourcetype=* | fields host, sourcetype | sort host | cluster | stats count by host, sourcetype | fields host,sourcetype

[sowings]

The data in that panel is coming from a search starting with |metadata type=host. This metadata is not simultaneously differentiated by host and by sourcetype (the left panel is sourcetype, ignoring host, the right panel is host, ignoring sourcetype). I don't know offhand why you're not seeing per_sourcetype_thruput in the forwarder's logs, perhaps it doesn't get logged there.

[reed_kelly]

I think you are referring to the Summary view in the Search app. This shows counts by sourcetype, but I want count by sourcetype,

[sowings]

If the heavy forwarder is sending over its own _internal indexing data (it doesn't by default, check forwardedindex.* in outputs.conf), you could utilize a metrics search (group=per_sourcetype_thruput) like the following:

index=_internal source=*metrics.log group=per_sourcetype_thruput | stats count by host, series

You'll get some of the "internal" sourcetypes which map to data that goes to the _internal index, like "audittrail" and "scheduler", etc, but it's a start.

[reed_kelly]

I tried that, but
grep sourcetype $SPLUNK_HOME/var/log/splunk/metric*
doesn't return anything on the forwarder.

[sowings]

It's not doing any local indexing, but by default it's also not forwarding anything which would have landed in its _internal index (like the metrics.log) either. In any event, you might consider reading through the metrics.log directly on the forwarder itself ($SPLUNK_HOME/var/log/splunk). Instead of a Splunk search, you'll be doing a grep (for group=per_sourcetype_thruput) and some awk. My point about forwarding _internal data from the forwarder would still be useful in situations like this.

[reed_kelly]

Thanks for your answer, but I think I wasn't clear enough. The heavy forwarder is not doing any local indexing. I want to know how much of each sourcetype it is sending on it's tcpout stream.

[MuS]

Hi reed.kelly

haven't checked it yet, but isn't there some saved search like that in the deployment app?

cheers

[reed_kelly]

I see total volume for each forwarder, but not a breakdown of the sourcetypes coming from each forwarder.