topic Re: Why would thruput metrics be missing from some forwarders? in Getting Data In

Why would thruput metrics be missing from some forwarders?

gabriel_vasseur — Wed, 06 Mar 2019 16:12:55 GMT

I'm using the following search to measure throughput of each forwarder:

index=_internal sourcetype=splunkd metrics source=*metrics.log group=per_sourcetype_thruput | stats sum(kb) as kb by host

This works well generally, but there is one forwarder for which this gives nothing. I really don't understand why.

We do get its metrics logs in _internal, but they have no thruput logs. The docs say:

Thruput is measured in the indexing pipeline. If your data is not reaching this pipeline for some reason, it will not appear in this data.

I don't understand what that means. The forwarder in question is a universal forwarder. We have very similar forwarders for which we have no problem, so I can't really understand what's different.

Can anyone shed some light on this?

Re: Why would thruput metrics be missing from some forwarders?

nickhills — Wed, 06 Mar 2019 16:33:38 GMT

If a forwarder is under pressure (poor network quality, hitting throughput rate limiting, or generally swamped) It will prioritize your monitored data over sending its own _internal logs, which includes metrics etc.

This can also happen if the indexing tier is under similar pressure, but since you only have one host with this issue, that seems less likely.

Re: Why would thruput metrics be missing from some forwarders?

gabriel_vasseur — Tue, 26 Mar 2019 09:11:24 GMT

Interesting, but like you said unlikely to be the case here. We have PS on site this week so if there is time I'll ask them to take a look and will update this question if anything transpires.!