topic Re: Missing per_*_thruput metrics on 9.3.x Universal forwarders. in Getting Data In

Missing per_*_thruput metrics on 9.3.x Universal forwarders.

hrawat — Tue, 19 Nov 2024 01:50:33 GMT

Apply following workaround in default-mode.conf

Additionally you can also push this change via DS push across thousands of universal forwarders.

Add index_thruput in the list of disabled processors.

Add following line as is in default-mode.conf.

#Turn off a processor [pipeline:indexerPipe] disabled_processors= index_thruput, indexer, indexandforward, latencytracker, diskusage, signing,tcp-output-generic-processor, syslog-output-generic-processor, http-output-generic-processor, stream-output-processor, s2soverhttpoutput, destination-key-processor

NOTE: PLEASE DON'T APPLY ON HF/SH/IDX/CM/DS. You want to use different app( not SplunkUniversalForwarder app) to push the change.

Re: Missing per_*_thruput metrics on 9.3.x Universal forwarders.

gjanders — Sun, 27 Oct 2024 22:44:42 GMT

Thanks for the information, I assume the target is to fix this in a future UF 9.3.x release?

Furthermore, would you happen to know what would happen if the setting was accidentally applied on a HF?

Clients of our deployment server will sometimes run a Splunk enterprise version instead of a UF so I suspect we will need to be careful...

Re: Missing per_*_thruput metrics on 9.3.x Universal forwarders.

hrawat — Sun, 27 Oct 2024 23:18:40 GMT

Applying on non-UF (e.g HF) will break thruput metrics. Added warning to post. Thanks for asking great question.

Re: Missing per_*_thruput metrics on 9.3.x Universal forwarders.

hrawat — Sun, 27 Oct 2024 23:24:10 GMT

https://community.splunk.com/t5/Getting-Data-In/Missing-per-thruput-metrics-on-9-3-x-Universal-forwarders/m-p/702914/highlight/true#M116255

Re: Missing per_*_thruput metrics on 9.3.x Universal forwarders.

d16 — Wed, 22 Jan 2025 15:32:03 GMT

I am a bit confused on the guidance here...

Does this re-enable the log(s) ?

We use the file /opt/splunkforwarder/var/log/splunk/metrics.log to check on our linux UF deploys that the /var/log/messages and auditd are appearing to send with some basic foo in our deploy scripts. With the SPL-263518 this is disabled by default now and we either need to identify another method of a simple local check or we need to re-enable group=per_source_thruput so we can rely on that check

sudo grep -c /opt/splunkforwarder/var/log/splunk/metrics.log -e 'INFO Metrics - group=per_source_thruput, series="/var/log/messages", kbps=') -ne 0

Is there a full writeup on SPL-263518 that has more info than the simple blurb on known-issues starting with 9.3.x? aka: was this removed for a security reason or just simply to reduce local log writes, etc?

Re: Missing per_*_thruput metrics on 9.3.x Universal forwarders.

hrawat — Wed, 22 Jan 2025 17:59:08 GMT

>Does this re-enable the log(s) ?
Yes

>we need to re-enable group=per_source_thruput so we can rely on that check

Apply the workaround.

>was this removed for a security reason or just simply to reduce local log writes, etc?

Accidentally got removed( regression)

Re: Missing per_*_thruput metrics on 9.3.x Universal forwarders.

d16 — Wed, 22 Jan 2025 18:03:04 GMT

Ah ok - that helpful info. the SPL-263518 on both 9.3 and 9.4 releases doesnt really state it was a regression and no link there explaining that...would be easier as a consumer if that SPL linked to a longer writeup/explanation.

Do you happen to know if there a plan/timeline for re-adding it?

Will it go into like 9.3.3 and 9.4.1 or will 9.3 and 9.4 just keep this regression and then 9.5 will re-add perhaps?

Re: Missing per_*_thruput metrics on 9.3.x Universal forwarders.

hrawat — Fri, 28 Mar 2025 15:25:47 GMT

9.5/10.0 (depending on actual future version) has the fix. Meaning the functionality is restored.
Not backported for 9.3.x/9.4.x.

Re: Missing per_*_thruput metrics on 9.3.x Universal forwarders.

hrawat — Fri, 18 Apr 2025 10:42:03 GMT

Note: As a side effect of this issue, maxKbps(limits.conf) will also be impacted as it requires thruput metrics to function.

Re: Missing per_*_thruput metrics on 9.3.x Universal forwarders.

jstratton — Mon, 21 Apr 2025 19:00:47 GMT

@hrawat wrote:
Note: As a side effect of this issue, maxKbps(limits.conf) will also be impacted as it requires thruput metrics to function.

Can you elaborate on how maxKbps is impacted?

Re: Missing per_*_thruput metrics on 9.3.x Universal forwarders.

hrawat — Mon, 21 Apr 2025 21:17:34 GMT

maxKbps is calculated from name=thruput. Since it's missing, so maxKbps is not working/applied.

Re: Missing per_*_thruput metrics on 9.3.x Universal forwarders.

jstratton — Mon, 21 Apr 2025 21:33:37 GMT

@hrawat wrote:
maxKbps is calculated from name=thruput. Since it's missing, so maxKbps is not working/applied.

Thx. Splunk is certain they will not back port the fix to 9.3.x and 9.4.x? Having per_*_thruput *and* maxKbps broken w/o the workaround seems worthy of a back port. Or at the very least, the "Known Issues" for SPL-263518 should be updated to mention maxKbps not working / applied.

Re: Missing per_*_thruput metrics on 9.3.x Universal forwarders.

hrawat — Mon, 21 Apr 2025 21:38:06 GMT

maxKbps was reported few days ago and it will be updated to known issues as well.

Re: Missing per_*_thruput metrics on 9.3.x Universal forwarders.

hrawat — Mon, 21 Apr 2025 21:53:06 GMT

>maxKbps broken w/o the workaround

Same workaround for maxKbps as well.

#Turn off a processor
[pipeline:indexerPipe]
disabled_processors= index_thruput, indexer, indexandforward, latencytracker, diskusage, signing,tcp-output-generic-processor, syslog-output-generic-processor, http-output-generic-processor, stream-output-processor, s2soverhttpoutput, destination-key-processor

Re: Missing per_*_thruput metrics on 9.3.x Universal forwarders.

sborys93 — Thu, 24 Apr 2025 14:06:00 GMT

Just to confirm here. When we say.

"Note: As a side effect of this issue, maxKbps(limits.conf) will also be impacted as it requires thruput metrics to function."

Are we saying that the following parameter in limits.conf is no longer applied/valid when modified?

[thruput]

maxKBps

I originally thought this solely a regression on the thruput maxKBps metric not being displayed in the logs.

Re: Missing per_*_thruput metrics on 9.3.x Universal forwarders.

hrawat — Thu, 24 Apr 2025 14:42:43 GMT

>Are we saying that the following parameter in limits.conf is no longer applied/valid when modified?
Yes on UF.

Re: Missing per_*_thruput metrics on 9.3.x Universal forwarders.

sborys93 — Fri, 12 Sep 2025 15:38:51 GMT

Just so it's known. There is a difference between placing this in

~/etc/apps/SplunkUniversalForwarder/default/default-mode.conf

vs

~/etc/system/default/default-mode.conf

You want to place this in ~/etc/apps/SplunkUniversalForwarder/default/default-mode.conf, otherwise it doesn't work.

Re: Missing per_*_thruput metrics on 9.3.x Universal forwarders.

hrawat — Fri, 12 Sep 2025 15:45:10 GMT

You should not make any changes in any default ( ~/etc/system/default/default-mode.conf/~/etc/apps/SplunkUniversalForwarder/default/default-mode.conf) folders.

Either in

~/etc/system/local/default-mode.conf/ Or ~/etc/apps/SplunkUniversalForwarder/local/default-mode.conf