Hi @kate,
You can enable the introspection generator add-on on forwarders by following the process at https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/ConfigurePIF#Enable_the_introspe.... If you're not using a deployment server, you can enable the add-on locally on any forwarder.
Note that the SplunkForwarder service account, e.g. NT SERVICE\SplunkForwarder, must have the "Debug programs" (SeDebugPrivilege) user right. While this isn't equivalent to administrator privileges, it does grant the user the ability to inject arbitrary code into another process running with administrator privileges. You can find more information in Microsoft security documentation. Don't fear the privilege, though. Just understand what it does and how to mitigate the risk of assigning in the context of Splunk.
By default, introspection:generator:resource_usage will be enabled and collect metrics every 10 minutes when the add-on is enabled is enabled on universal forwarders.
You can find metrics in index=_introspection, an event index containing source types with INDEXED_EXTRACTIONS = json:
| tstats avg(data.cpu_idle_pct) as cpu_idle_pct where index=_introspection sourcetype=splunk_resource_usage component=Hostwide by _time host
| chart avg(eval(100-cpu_idle_pct)) ``` cpu_used_pct ``` over _time by host
On instances of Splunk Enterprise, metrics are also cloned to index=_metrics; however, events sent from forwarders with INDEXED_EXTRACTIONS set are "cooked" by the forwarder, and transforms on receivers will not be applied without modifying configuration to reroute cooked events to parsingQueue or adding ingest actions (rulesets) that reference the transforms behavior.
