topic Re: New Splunk Metrics logging interval in Knowledge Management

New Splunk Metrics logging interval

hrawat — Tue, 26 Nov 2024 21:20:27 GMT

Starting 9.2.0 release internal metrics log event generation can be controlled by group or subgroup. If there are thousands of forwarders, _internal index becomes the most active index and generates lot of hot buckets.
9.2.0 provides ability to control each metrics group/subgroup.

checkout `interval` https://docs.splunk.com/Documentation/Splunk/latest/Admin/limitsconf for more details.

Upon splunk start, metrics.log will log all the controllable metrics groups/subgroups.
Example metrics.log.

06-08-2024 03:14:49.659 +0000 INFO Metrics - Will log metrics_module=dutycycle:ingest at metrics_interval=30.000.

metrics_module is the controllable module logged in metrics.log. dutycycle:ingest is the the controllable module. dutycycle is the metrics group name and ingest is the subgroup name. It's default logging interval is 30 sec.

06-08-2024 03:14:49.703 +0000 INFO Metrics - Will log metrics_module=tailingprocessor:tailreader0 at metrics_interval=60.000.

tailingprocessor is the group name and tailreader0 is the subgroup name( where trailing `0` is the first pipeline number). It's default logging interval is 60 sec.

New metrics logging framework has global default metrics logging interval 60 sec in limits.conf with exception for some modules(30 sec) you will find in metrics.log

[metrics] interval = <integer> * Number of seconds between logging splunkd metrics to metrics.log. * Minimum of 10. * Default (Splunk Enterprise): 60 * Default (Splunk Universal Forwarder): 60

There are so many modules you will find in metrics.log that are never queried. For example queue and thruput metrics is probably the most queried metrics but not necessarily others.

You can increase global default to 120 second

[metrics] interval=120

Customize logging interval for other very critical metrics groups.
Example there are various `queue` metrics logging. Some are always checked, some rarely.

06-08-2024 03:14:49.603 +0000 INFO Metrics - Will log metrics_module=queue:parsingqueue at metrics_interval=30.000. 06-08-2024 03:14:49.663 +0000 INFO Metrics - Will log metrics_module=queue:httpinputq at metrics_interval=30.000. 06-08-2024 03:14:49.651 +0000 INFO Metrics - Will log metrics_module=queue:stashparsing at metrics_interval=30.000. 06-08-2024 03:14:49.603 +0000 INFO Metrics - Will log metrics_module=queue:teequeue at metrics_interval=30.000.

You can set global default for queue group.

[queue] interval=60

parsing 30 sec.

[queue:parsingqueue] interval=30

stashparsing 150 sec.

[queue:stashparsing] interval=150

Interval can be set for [<group>] or [<group>:<subgroup>].

Re: New Splunk Metrics logging interval

psla — Fri, 12 Jul 2024 12:01:59 GMT

Why isn't it listed under "What's new"?

https://docs.splunk.com/Documentation/Splunk/9.2.2/ReleaseNotes/MeetSplunk

Re: New Splunk Metrics logging interval

hrawat — Tue, 26 Nov 2024 21:18:48 GMT

Not one of the change to qualify as new, as it's rather an improvement.

It's documented in limits.conf.

interval = <integer> * Number of seconds between logging splunkd metrics to metrics.log for different subgroups. * Check metrics.log for the list of configurable "metrics_modules". * Set "interval" under the desired "metrics_module" stanza. * Example: * If you want 60 seconds metrics logging interval for "thruput:thruput", * [thruput:thruput] * interval = 60 * Minimum value is 10 seconds. * Valid value is multiple of 10. * If value is not exact multiple of 10, it will be adjusted to nearest downward multiple. * Recommended value multiple of 30. Splunk will decide how often to check for metrics reporting based on greatest common divisor across different values. If "interval" is set 30, 40 for two different components, then greatest common divisor for 30, 40 and 60(default) is 10. It's expensive for metrics reporting thread to log every 10 sec. If "interval" is set 30, 900 for two different components, then greatest common divisor for 30, 90 and 60(default) is 30. It's less expensive for metrics reporting thread to log every 30 sec. * Default : "interval" config value set under [metrics] stanza.

https://docs.splunk.com/Documentation/Splunk/latest/Admin/limitsconf

Re: New Splunk Metrics logging interval

hrawat — Tue, 26 Nov 2024 22:32:11 GMT

https://community.splunk.com/t5/Knowledge-Management/New-Splunk-Metrics-logging-interval/m-p/705328/highlight/true#M10338