Getting Data In

Breakdown by Sourcetype from Each Forwarder

reed_kelly
Contributor

We have a number of heavy forwarders sending cooked data to our indexers. We can get the total KBs sent by each forwarder by searching the metrics.log files. Is there a way to get a breakdown by sourcetype of data sent by each forwarder? Maybe this is a search that we can run from each forwarder?

0 Karma

rgcurry
Contributor

In the Search app, the third panel down, the panel on the left displays all the sourcetypes and some data about them. Is this what you seek?

0 Karma

rgcurry
Contributor

Does this give you what you are looking for?

index=* sourcetype=* | fields host, sourcetype | sort host | cluster | stats count by host, sourcetype | fields host,sourcetype

0 Karma

sowings
Splunk Employee
Splunk Employee

The data in that panel is coming from a search starting with |metadata type=host. This metadata is not simultaneously differentiated by host and by sourcetype (the left panel is sourcetype, ignoring host, the right panel is host, ignoring sourcetype). I don't know offhand why you're not seeing per_sourcetype_thruput in the forwarder's logs, perhaps it doesn't get logged there.

0 Karma

reed_kelly
Contributor

I think you are referring to the Summary view in the Search app. This shows counts by sourcetype, but I want count by sourcetype,

0 Karma

sowings
Splunk Employee
Splunk Employee

If the heavy forwarder is sending over its own _internal indexing data (it doesn't by default, check forwardedindex.* in outputs.conf), you could utilize a metrics search (group=per_sourcetype_thruput) like the following:

index=_internal source=*metrics.log group=per_sourcetype_thruput | stats count by host, series

You'll get some of the "internal" sourcetypes which map to data that goes to the _internal index, like "audittrail" and "scheduler", etc, but it's a start.

reed_kelly
Contributor

I tried that, but
grep sourcetype $SPLUNK_HOME/var/log/splunk/metric*
doesn't return anything on the forwarder.

0 Karma

sowings
Splunk Employee
Splunk Employee

It's not doing any local indexing, but by default it's also not forwarding anything which would have landed in its _internal index (like the metrics.log) either. In any event, you might consider reading through the metrics.log directly on the forwarder itself ($SPLUNK_HOME/var/log/splunk). Instead of a Splunk search, you'll be doing a grep (for group=per_sourcetype_thruput) and some awk. My point about forwarding _internal data from the forwarder would still be useful in situations like this.

0 Karma

reed_kelly
Contributor

Thanks for your answer, but I think I wasn't clear enough. The heavy forwarder is not doing any local indexing. I want to know how much of each sourcetype it is sending on it's tcpout stream.

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi reed.kelly

haven't checked it yet, but isn't there some saved search like that in the deployment app?

cheers

0 Karma

reed_kelly
Contributor

I see total volume for each forwarder, but not a breakdown of the sourcetypes coming from each forwarder.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...