How can I calculate CPU of the splunk server in percentage from the data in internal index?
The data in internal index is as below where source = /opt/splunk/var/log/splunk/metrics.log
01-25-2024 15:47:42.528 +0000 INFO Metrics - group=pipeline, name=dev-null, processor=nullqueue, cpu_seconds=0.001, executes=4445, cumulative_hits=9717713
01-25-2024 15:47:42.527 +0000 INFO Metrics - group=workload_management, name=workload-statistics, workload_pool=standard_perf, mem_limit_in_bytes=71715885056, cpu_shares=358
01-25-2024 15:47:42.525 +0000 INFO Metrics - group=conf, action=acquire_mutex, count=20, wallclock_ms_total=0, wallclock_ms_max=0, cpu_total=0.000, cpu_max=0.000
Thank you for your response.
Is there are difference between the performance (CPU and memory) data for a UF in internal logs and the logs fetched by splunk add on for windows or splunk add on for unix and linux machine?
Internally, both the introspection generator and the tools used by the add-ons may use the same operating system interfaces to collect performance metrics; however, two observers running concurrently on the same host will return different values for the same metric. This is the nature of statistical sampling.
Hi @kate ,
yes, the Splunk_TA_Windows gives you more counters than internal logs, even if it consumes license.
Ciao.
Giuseppe
Hi @kate,
You can enable the introspection generator add-on on forwarders by following the process at https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/ConfigurePIF#Enable_the_introspe.... If you're not using a deployment server, you can enable the add-on locally on any forwarder.
Note that the SplunkForwarder service account, e.g. NT SERVICE\SplunkForwarder, must have the "Debug programs" (SeDebugPrivilege) user right. While this isn't equivalent to administrator privileges, it does grant the user the ability to inject arbitrary code into another process running with administrator privileges. You can find more information in Microsoft security documentation. Don't fear the privilege, though. Just understand what it does and how to mitigate the risk of assigning in the context of Splunk.
By default, introspection:generator:resource_usage will be enabled and collect metrics every 10 minutes when the add-on is enabled is enabled on universal forwarders.
You can find metrics in index=_introspection, an event index containing source types with INDEXED_EXTRACTIONS = json:
| tstats avg(data.cpu_idle_pct) as cpu_idle_pct where index=_introspection sourcetype=splunk_resource_usage component=Hostwide by _time host
| chart avg(eval(100-cpu_idle_pct)) ``` cpu_used_pct ``` over _time by host
On instances of Splunk Enterprise, metrics are also cloned to index=_metrics; however, events sent from forwarders with INDEXED_EXTRACTIONS set are "cooked" by the forwarder, and transforms on receivers will not be applied without modifying configuration to reroute cooked events to parsingQueue or adding ingest actions (rulesets) that reference the transforms behavior.
Hi @kate,
did you tried to use the Monitoring Console?
You already have all the dashboards you need.
for more information see at https://docs.splunk.com/Documentation/Splunk/9.1.2/DMC/DMCoverview
Ciao.
Giuseppe
I am looking forward to utilize only splunk internal logs for the same.
How can I utilize splunk internal metric log of a UF to fetch CPU and memory data for the same UF?
Hi @kate,
surely you are using an add on for your Universal Forwarder (Linux or Windows), in this case, you have to enable the cpu counter metrics in this add-on, then you can use these data to calculate percentage use.
Ciao.
Giuseppe