Getting Data In

Missing per_*_thruput metrics on 9.3.x Universal forwarders.

hrawat
Splunk Employee
Splunk Employee

Apply following workaround in default-mode.conf

Additionally you can also push this change via DS push across thousands of universal forwarders.

Add index_thruput in the list of disabled processors. 

Add following line as is in default-mode.conf.

 

#Turn off a processor
[pipeline:indexerPipe]
disabled_processors= index_thruput, indexer, indexandforward, latencytracker, diskusage, signing,tcp-output-generic-processor, syslog-output-generic-processor, http-output-generic-processor, stream-output-processor, s2soverhttpoutput, destination-key-processor

 

 

NOTE:  PLEASE DON'T APPLY ON HF/SH/IDX/CM/DS. You want to use different app( not SplunkUniversalForwarder app) to push the change.


Labels (1)

d16
Engager

I am a bit confused on the guidance here...

Does this re-enable the log(s) ? 

We use the file /opt/splunkforwarder/var/log/splunk/metrics.log to check on our linux UF deploys that the /var/log/messages and auditd are appearing to send with some basic foo in our deploy scripts. With the SPL-263518 this is disabled by default now and we either need to identify another method of a simple local check or we need to re-enable group=per_source_thruput so we can rely on that check

sudo grep -c /opt/splunkforwarder/var/log/splunk/metrics.log -e 'INFO  Metrics - group=per_source_thruput, series="/var/log/messages", kbps=') -ne 0

 

Is there a full writeup on SPL-263518 that has more info than the simple blurb on known-issues starting with 9.3.x? aka: was this removed for a security reason or just simply to reduce local log writes, etc? 

0 Karma

hrawat
Splunk Employee
Splunk Employee

>Does this re-enable the log(s) ? 
Yes

>we need to re-enable group=per_source_thruput so we can rely on that check

Apply the workaround.

>was this removed for a security reason or just simply to reduce local log writes, etc? 

Accidentally got removed( regression)

d16
Engager

Ah ok - that helpful info. the SPL-263518 on both 9.3 and 9.4 releases doesnt really state it was a regression and no link there explaining that...would be easier as a consumer if that SPL linked to a longer writeup/explanation.

Do you happen to know if there a plan/timeline for re-adding it?

Will it go into like 9.3.3 and 9.4.1 or will 9.3 and 9.4 just keep this regression and then 9.5 will re-add perhaps?

0 Karma

hrawat
Splunk Employee
Splunk Employee

9.5 has the fix. Meaning the functionality is restored.
Not backported for 9.3.x/9.4.x.  

hrawat
Splunk Employee
Splunk Employee

Applying on non-UF (e.g HF) will break thruput metrics. Added warning to post. Thanks for asking great question.

gjanders
SplunkTrust
SplunkTrust

Thanks for the information, I assume the target is to fix this in a future UF 9.3.x release?

Furthermore, would you happen to know what would happen if the setting was accidentally applied on a HF?

 

Clients of our deployment server will sometimes run a Splunk enterprise version instead of a UF so I suspect we will need to be careful...

0 Karma
Get Updates on the Splunk Community!

Message Parsing in SOCK

Introduction This blog post is part of an ongoing series on SOCK enablement. In this blog post, I will write ...

Exploring the OpenTelemetry Collector’s Kubernetes annotation-based discovery

We’ve already explored a few topics around observability in a Kubernetes environment -- Common Failures in a ...

Use ‘em or lose ‘em | Splunk training units do expire

Whether it’s hummus, a ham sandwich, or a human, almost everything in this world has an expiration date. And, ...