New Splunk Metrics logging interval

[hrawat]

Starting 9.2.0 release internal metrics log event generation can be controlled by group or subgroup.  If there are thousands of forwarders,  _internal index becomes the most active index and generates lot of hot buckets.
9.2.0 provides ability to control each metrics group/subgroup.

checkout `interval` https://docs.splunk.com/Documentation/Splunk/latest/Admin/limitsconf for more details.

Upon splunk start, metrics.log will log all the controllable metrics groups/subgroups. 
Example metrics.log.

06-08-2024 03:14:49.659 +0000 INFO  Metrics - Will log metrics_module=dutycycle:ingest at metrics_interval=30.000.

metrics_module is the controllable module logged in metrics.log. dutycycle:ingest is the the controllable module. dutycycle is the metrics group name and ingest is the subgroup name. It's default logging interval is 30 sec.

06-08-2024 03:14:49.703 +0000 INFO  Metrics - Will log metrics_module=tailingprocessor:tailreader0 at metrics_interval=60.000.

tailingprocessor is the group name and tailreader0 is the subgroup name( where trailing `0` is the first pipeline number). It's default logging interval is 60 sec.

New metrics logging framework has global default metrics logging interval 60 sec in limits.conf with exception for some modules(30 sec) you will find in metrics.log

[metrics]
interval = <integer>
* Number of seconds between logging splunkd metrics to metrics.log.
* Minimum of 10.
* Default (Splunk Enterprise): 60
* Default (Splunk Universal Forwarder): 60

There are so many modules you will find in metrics.log that are never queried. For example queue and thruput metrics is probably the most queried metrics but not necessarily others.

You can increase global default to 120 second

[metrics]
interval=120

Customize logging interval for other very critical metrics groups.
Example there are various `queue` metrics logging. Some are always checked, some rarely.

06-08-2024 03:14:49.603 +0000 INFO  Metrics - Will log metrics_module=queue:parsingqueue at metrics_interval=30.000.
06-08-2024 03:14:49.663 +0000 INFO  Metrics - Will log metrics_module=queue:httpinputq at metrics_interval=30.000.
06-08-2024 03:14:49.651 +0000 INFO  Metrics - Will log metrics_module=queue:stashparsing at metrics_interval=30.000.
06-08-2024 03:14:49.603 +0000 INFO  Metrics - Will log metrics_module=queue:teequeue at metrics_interval=30.000.

 You can set global default for queue group.

[queue]
interval=60

parsing 30 sec.

[queue:parsingqueue]
interval=30

stashparsing 150 sec.

[queue:stashparsing]
interval=150

 
Interval can be set for [<group>] or [<group>:<subgroup>].

[hrawat]

Not one of the change to qualify as new, as it's rather an improvement.

It's documented in limits.conf.

interval = <integer>
* Number of seconds between logging splunkd metrics to metrics.log
  for different subgroups.
* Check metrics.log for the list of configurable "metrics_modules".
* Set "interval" under the desired "metrics_module" stanza.
* Example:
  * If you want 60 seconds metrics logging interval for "thruput:thruput",
    * [thruput:thruput]
    * interval = 60
* Minimum value is 10 seconds.
* Valid value is multiple of 10.
* If value is not exact multiple of 10, it will be adjusted to nearest
  downward multiple.
* Recommended value multiple of 30. Splunk will decide how often to check for
  metrics reporting based on greatest common divisor across different values.
  If "interval" is set 30, 40 for two different components, then
  greatest common divisor for 30, 40 and 60(default) is 10. It's expensive
  for metrics reporting thread to log every 10 sec.
  If "interval" is set 30, 900 for two different components, then
  greatest common divisor for 30, 90 and 60(default) is 30. It's less
  expensive for metrics reporting thread to log every 30 sec.
* Default : "interval" config value set under [metrics] stanza.

https://docs.splunk.com/Documentation/Splunk/latest/Admin/limitsconf

[psla]

Why isn't it listed under "What's new"?

https://docs.splunk.com/Documentation/Splunk/9.2.2/ReleaseNotes/MeetSplunk

[hrawat]

https://community.splunk.com/t5/Knowledge-Management/New-Splunk-Metrics-logging-interval/m-p/705328/...