Missing per_*_thruput metrics on 9.3.x Universal forwarders.

[hrawat]

Apply following workaround in default-mode.conf

Additionally you can also push this change via DS push across thousands of universal forwarders.

Add index_thruput in the list of disabled processors. 

Add following line as is in default-mode.conf.

#Turn off a processor
[pipeline:indexerPipe]
disabled_processors= index_thruput, indexer, indexandforward, latencytracker, diskusage, signing,tcp-output-generic-processor, syslog-output-generic-processor, http-output-generic-processor, stream-output-processor, s2soverhttpoutput, destination-key-processor

NOTE:  PLEASE DON'T APPLY ON HF/SH/IDX/CM/DS. You want to use different app( not SplunkUniversalForwarder app) to push the change.

[sborys93]

Just so it's known. There is a difference between placing this in 

~/etc/apps/SplunkUniversalForwarder/default/default-mode.conf

vs

~/etc/system/default/default-mode.conf

You want to place this in ~/etc/apps/SplunkUniversalForwarder/default/default-mode.conf, otherwise it doesn't work.

[hrawat]

You should not make any changes in any default ( ~/etc/system/default/default-mode.conf/~/etc/apps/SplunkUniversalForwarder/default/default-mode.conf) folders.

Either in 

~/etc/system/local/default-mode.conf/
Or
~/etc/apps/SplunkUniversalForwarder/local/default-mode.conf

[sborys93]

Just to confirm here. When we say.

"Note: As a side effect of this issue, maxKbps(limits.conf) will also be impacted as it requires thruput metrics to function."

Are we saying that the following parameter in limits.conf is no longer applied/valid when modified?

[thruput]

maxKBps

I originally thought this solely a regression on the thruput maxKBps metric not being displayed in the logs.

[hrawat]

>Are we saying that the following parameter in limits.conf is no longer applied/valid when modified?
Yes on UF.

[hrawat]

Note: As a side effect of this issue, maxKbps(limits.conf) will also be impacted as it requires thruput metrics to function.

[jstratton]

---

@hrawat wrote:

Note: As a side effect of this issue, maxKbps(limits.conf) will also be impacted as it requires thruput metrics to function.

---

Can you elaborate on how maxKbps is impacted?

[hrawat]

maxKbps is calculated from name=thruput. Since it's missing, so maxKbps is not working/applied.

[jstratton]

---

@hrawat wrote:

maxKbps is calculated from name=thruput. Since it's missing, so maxKbps is not working/applied.

---

Thx. Splunk is certain they will not back port the fix to 9.3.x and 9.4.x? Having per_*_thruput *and* maxKbps broken w/o the workaround seems worthy of a back port. Or at the very least, the "Known Issues" for SPL-263518 should be updated to mention maxKbps not working / applied.

[hrawat]

>maxKbps broken w/o the workaround 

Same workaround for maxKbps as well.

#Turn off a processor
[pipeline:indexerPipe]
disabled_processors= index_thruput, indexer, indexandforward, latencytracker, diskusage, signing,tcp-output-generic-processor, syslog-output-generic-processor, http-output-generic-processor, stream-output-processor, s2soverhttpoutput, destination-key-processor

[hrawat]

maxKbps was reported few days ago and it will be updated to known issues as well.

[d16]

I am a bit confused on the guidance here...

Does this re-enable the log(s) ? 

We use the file /opt/splunkforwarder/var/log/splunk/metrics.log to check on our linux UF deploys that the /var/log/messages and auditd are appearing to send with some basic foo in our deploy scripts. With the SPL-263518 this is disabled by default now and we either need to identify another method of a simple local check or we need to re-enable group=per_source_thruput so we can rely on that check

sudo grep -c /opt/splunkforwarder/var/log/splunk/metrics.log -e 'INFO  Metrics - group=per_source_thruput, series="/var/log/messages", kbps=') -ne 0

Is there a full writeup on SPL-263518 that has more info than the simple blurb on known-issues starting with 9.3.x? aka: was this removed for a security reason or just simply to reduce local log writes, etc? 

[hrawat]

>Does this re-enable the log(s) ? 
Yes

>we need to re-enable group=per_source_thruput so we can rely on that check

Apply the workaround.

>was this removed for a security reason or just simply to reduce local log writes, etc? 

Accidentally got removed( regression)

[d16]

Ah ok - that helpful info. the SPL-263518 on both 9.3 and 9.4 releases doesnt really state it was a regression and no link there explaining that...would be easier as a consumer if that SPL linked to a longer writeup/explanation.

Do you happen to know if there a plan/timeline for re-adding it?

Will it go into like 9.3.3 and 9.4.1 or will 9.3 and 9.4 just keep this regression and then 9.5 will re-add perhaps?

[hrawat]

9.5/10.0 (depending on actual future version) has the fix. Meaning the functionality is restored.
Not backported for 9.3.x/9.4.x.  

[hrawat]

Applying on non-UF (e.g HF) will break thruput metrics. Added warning to post. Thanks for asking great question.

[gjanders]

Thanks for the information, I assume the target is to fix this in a future UF 9.3.x release?

Furthermore, would you happen to know what would happen if the setting was accidentally applied on a HF?

Clients of our deployment server will sometimes run a Splunk enterprise version instead of a UF so I suspect we will need to be careful...

[hrawat]

https://community.splunk.com/t5/Getting-Data-In/Missing-per-thruput-metrics-on-9-3-x-Universal-forwa...