Getting Data In

Missing per_*_thruput metrics on 9.3.x Universal forwarders.

hrawat
Splunk Employee
Splunk Employee

Apply following workaround in default-mode.conf

Additionally you can also push this change via DS push across thousands of universal forwarders.

Add index_thruput in the list of disabled processors. 

Add following line as is in default-mode.conf.

 

#Turn off a processor
[pipeline:indexerPipe]
disabled_processors= index_thruput, indexer, indexandforward, latencytracker, diskusage, signing,tcp-output-generic-processor, syslog-output-generic-processor, http-output-generic-processor, stream-output-processor, s2soverhttpoutput, destination-key-processor

 

 

NOTE:  PLEASE DON'T APPLY ON HF/SH/IDX/CM/DS. You want to use different app( not SplunkUniversalForwarder app) to push the change.


Labels (1)

sborys93
Engager

Just so it's known. There is a difference between placing this in 

~/etc/apps/SplunkUniversalForwarder/default/default-mode.conf

vs

~/etc/system/default/default-mode.conf

You want to place this in ~/etc/apps/SplunkUniversalForwarder/default/default-mode.conf, otherwise it doesn't work.

0 Karma

hrawat
Splunk Employee
Splunk Employee

You should not make any changes in any default ( ~/etc/system/default/default-mode.conf/~/etc/apps/SplunkUniversalForwarder/default/default-mode.conf) folders.

Either in 

~/etc/system/local/default-mode.conf/
Or
~/etc/apps/SplunkUniversalForwarder/local/default-mode.conf

sborys93
Engager

Just to confirm here. When we say.

"Note: As a side effect of this issue, maxKbps(limits.conf) will also be impacted as it requires thruput metrics to function."

Are we saying that the following parameter in limits.conf is no longer applied/valid when modified?

[thruput]

maxKBps


I originally thought this solely a regression on the thruput maxKBps metric not being displayed in the logs.

0 Karma

hrawat
Splunk Employee
Splunk Employee

>Are we saying that the following parameter in limits.conf is no longer applied/valid when modified?
Yes on UF.

hrawat
Splunk Employee
Splunk Employee

Note: As a side effect of this issue, maxKbps(limits.conf) will also be impacted as it requires thruput metrics to function.

0 Karma

jstratton
Explorer

@hrawat wrote:

Note: As a side effect of this issue, maxKbps(limits.conf) will also be impacted as it requires thruput metrics to function.


Can you elaborate on how maxKbps is impacted?

0 Karma

hrawat
Splunk Employee
Splunk Employee

maxKbps is calculated from name=thruput. Since it's missing, so maxKbps is not working/applied.

jstratton
Explorer

@hrawat wrote:

maxKbps is calculated from name=thruput. Since it's missing, so maxKbps is not working/applied.


Thx. Splunk is certain they will not back port the fix to 9.3.x and 9.4.x? Having per_*_thruput *and* maxKbps broken w/o the workaround seems worthy of a back port. Or at the very least, the "Known Issues" for SPL-263518 should be updated to mention maxKbps not working / applied.

0 Karma

hrawat
Splunk Employee
Splunk Employee

>maxKbps broken w/o the workaround 

Same workaround for maxKbps as well.

#Turn off a processor
[pipeline:indexerPipe]
disabled_processors= index_thruput, indexer, indexandforward, latencytracker, diskusage, signing,tcp-output-generic-processor, syslog-output-generic-processor, http-output-generic-processor, stream-output-processor, s2soverhttpoutput, destination-key-processor

  

0 Karma

hrawat
Splunk Employee
Splunk Employee

maxKbps was reported few days ago and it will be updated to known issues as well.

d16
Engager

I am a bit confused on the guidance here...

Does this re-enable the log(s) ? 

We use the file /opt/splunkforwarder/var/log/splunk/metrics.log to check on our linux UF deploys that the /var/log/messages and auditd are appearing to send with some basic foo in our deploy scripts. With the SPL-263518 this is disabled by default now and we either need to identify another method of a simple local check or we need to re-enable group=per_source_thruput so we can rely on that check

sudo grep -c /opt/splunkforwarder/var/log/splunk/metrics.log -e 'INFO  Metrics - group=per_source_thruput, series="/var/log/messages", kbps=') -ne 0

 

Is there a full writeup on SPL-263518 that has more info than the simple blurb on known-issues starting with 9.3.x? aka: was this removed for a security reason or just simply to reduce local log writes, etc? 

0 Karma

hrawat
Splunk Employee
Splunk Employee

>Does this re-enable the log(s) ? 
Yes

>we need to re-enable group=per_source_thruput so we can rely on that check

Apply the workaround.

>was this removed for a security reason or just simply to reduce local log writes, etc? 

Accidentally got removed( regression)

d16
Engager

Ah ok - that helpful info. the SPL-263518 on both 9.3 and 9.4 releases doesnt really state it was a regression and no link there explaining that...would be easier as a consumer if that SPL linked to a longer writeup/explanation.

Do you happen to know if there a plan/timeline for re-adding it?

Will it go into like 9.3.3 and 9.4.1 or will 9.3 and 9.4 just keep this regression and then 9.5 will re-add perhaps?

0 Karma

hrawat
Splunk Employee
Splunk Employee

9.5/10.0 (depending on actual future version) has the fix. Meaning the functionality is restored.
Not backported for 9.3.x/9.4.x.  

hrawat
Splunk Employee
Splunk Employee

Applying on non-UF (e.g HF) will break thruput metrics. Added warning to post. Thanks for asking great question.

gjanders
SplunkTrust
SplunkTrust

Thanks for the information, I assume the target is to fix this in a future UF 9.3.x release?

Furthermore, would you happen to know what would happen if the setting was accidentally applied on a HF?

 

Clients of our deployment server will sometimes run a Splunk enterprise version instead of a UF so I suspect we will need to be careful...

0 Karma
Get Updates on the Splunk Community!

Fall Into Learning with New Splunk Education Courses

Every month, Splunk Education releases new courses to help you branch out, strengthen your data science roots, ...

Super Optimize your Splunk Stats Searches: Unlocking the Power of tstats, TERM, and ...

By Martin Hettervik, Senior Consultant and Team Leader at Accelerate at Iver, Splunk MVPThe stats command is ...

How Splunk Observability Cloud Prevented a Major Payment Crisis in Minutes

Your bank's payment processing system is humming along during a busy afternoon, handling millions in hourly ...