topic Re: Breakdown by Sourcetype from Each Forwarder in Getting Data In

Breakdown by Sourcetype from Each Forwarder

reed_kelly — Tue, 24 Apr 2012 13:30:22 GMT

We have a number of heavy forwarders sending cooked data to our indexers. We can get the total KBs sent by each forwarder by searching the metrics.log files. Is there a way to get a breakdown by sourcetype of data sent by each forwarder? Maybe this is a search that we can run from each forwarder?

Re: Breakdown by Sourcetype from Each Forwarder

MuS — Tue, 24 Apr 2012 14:17:36 GMT

Hi reed.kelly

haven't checked it yet, but isn't there some saved search like that in the deployment app?

cheers

Re: Breakdown by Sourcetype from Each Forwarder

reed_kelly — Tue, 24 Apr 2012 14:25:34 GMT

I see total volume for each forwarder, but not a breakdown of the sourcetypes coming from each forwarder.

Re: Breakdown by Sourcetype from Each Forwarder

sowings — Mon, 28 Sep 2020 11:42:52 GMT

If the heavy forwarder is sending over its own _internal indexing data (it doesn't by default, check forwardedindex.* in outputs.conf), you could utilize a metrics search (group=per_sourcetype_thruput) like the following:

index=_internal source=*metrics.log group=per_sourcetype_thruput | stats count by host, series

You'll get some of the "internal" sourcetypes which map to data that goes to the _internal index, like "audittrail" and "scheduler", etc, but it's a start.

Re: Breakdown by Sourcetype from Each Forwarder

reed_kelly — Tue, 24 Apr 2012 15:59:26 GMT

Thanks for your answer, but I think I wasn't clear enough. The heavy forwarder is not doing any local indexing. I want to know how much of each sourcetype it is sending on it's tcpout stream.

Re: Breakdown by Sourcetype from Each Forwarder

sowings — Mon, 28 Sep 2020 11:43:02 GMT

It's not doing any local indexing, but by default it's also not forwarding anything which would have landed in its _internal index (like the metrics.log) either. In any event, you might consider reading through the metrics.log directly on the forwarder itself ($SPLUNK_HOME/var/log/splunk). Instead of a Splunk search, you'll be doing a grep (for group=per_sourcetype_thruput) and some awk. My point about forwarding _internal data from the forwarder would still be useful in situations like this.

Re: Breakdown by Sourcetype from Each Forwarder

reed_kelly — Tue, 24 Apr 2012 17:39:59 GMT

I tried that, but
grep sourcetype $SPLUNK_HOME/var/log/splunk/metric*
doesn't return anything on the forwarder.

Re: Breakdown by Sourcetype from Each Forwarder

rgcurry — Tue, 24 Apr 2012 17:44:44 GMT

In the Search app, the third panel down, the panel on the left displays all the sourcetypes and some data about them. Is this what you seek?

Re: Breakdown by Sourcetype from Each Forwarder

reed_kelly — Tue, 24 Apr 2012 17:51:20 GMT

I think you are referring to the Summary view in the Search app. This shows counts by sourcetype, but I want count by sourcetype,

Re: Breakdown by Sourcetype from Each Forwarder

sowings — Tue, 24 Apr 2012 18:23:59 GMT

The data in that panel is coming from a search starting with |metadata type=host. This metadata is not simultaneously differentiated by host and by sourcetype (the left panel is sourcetype, ignoring host, the right panel is host, ignoring sourcetype). I don't know offhand why you're not seeing per_sourcetype_thruput in the forwarder's logs, perhaps it doesn't get logged there.

Re: Breakdown by Sourcetype from Each Forwarder

rgcurry — Tue, 24 Apr 2012 20:40:07 GMT

Does this give you what you are looking for?