UPDATE: This procedure is now officially documented here.
In order to be enable a built-in app such as "introspection_generator_addon" on a deployment client, your deployment server needs to be upgraded to Splunk Enterprise 6.2, which introduced the excludeFromUpdate option to serverclass.conf. From serverclass.conf.spec:
excludeFromUpdate = <path>[,<path>]...
* Specifies paths to one or more top-level files or directories (and their contents) to exclude from being touched during app update. Note that each comma-separated entry MUST be prefixed by "$app_root$/" (otherwise a warning will be generated).
So, the idea here is that in order to turn on this built-in app, your deployment server needs to ship a version of it that only contains an app.conf file in the "local" directory, with the state = enabled directive. All other directories in the app would need to be excluded from update.
Here's an example of how that can be achieved:
Deployment server, simplified serverclass.conf:
[global]
whitelist.0=*
[serverClass:AllApps]
[serverClass:AllApps:app:introspection_generator_addon]
excludeFromUpdate = $app_root$/default, $app_root$/bin
restartSplunkd = True
Note that we are fine-tuning what is being pushed at the app level and excluding both the "bin" and "default" directories. We ONLY want to push stuff to the "local" directory for this app - namely, an app.conf that enables it on the forwarder:
[root@sosdev-sh:/opt/cm/splunk]# ls -lR etc/deployment-apps/introspection_generator_addon/
etc/deployment-apps/introspection_generator_addon/:
total 4
drwxr-xr-x. 2 root root 4096 Dec 5 22:43 local
etc/deployment-apps/introspection_generator_addon/local:
total 4
-rw-r--r--. 1 root root 26 Dec 5 22:43 app.conf
Contents of app.conf:
[root@sosdev-sh:/opt/cm/splunk]# cat etc/deployment-apps/introspection_generator_addon/local/app.conf
[install]
state = enabled
That's it - that's all that our deployment app contains: a local/app.conf file that enables the app.
On the Deployment client, post-deployment:
[root@sosdev-idx3 apps]# ls -lR introspection_generator_addon/
introspection_generator_addon/:
total 16
drwxr-xr-x. 2 506 506 4096 Oct 22 17:05 bin
drwxr-xr-x. 2 506 506 4096 Oct 22 17:05 default
drwx------. 2 root root 4096 Dec 5 23:18 local
drwx------. 2 root root 4096 Dec 5 23:18 metadata
introspection_generator_addon/bin:
total 4
-r-xr-xr-x. 1 506 506 53 Oct 22 16:32 collector.path
introspection_generator_addon/default:
total 16
-r--r--r--. 1 506 506 216 Oct 22 17:07 app.conf
-r--r--r--. 1 506 506 180 Oct 22 16:34 inputs.conf
-r--r--r--. 1 506 506 691 Oct 22 16:32 README
-r--r--r--. 1 506 506 560 Oct 22 16:34 server.conf
introspection_generator_addon/local:
total 4
-rw-------. 1 root root 26 Dec 5 23:18 app.conf
introspection_generator_addon/metadata:
total 4
-rw-------. 1 root root 67 Dec 5 23:18 local.meta
Note the presence of local/app.conf, as expected. Everything else is still there and untouched.
[root@sosdev-idx3 apps]# /opt/fwd5/splunkforwarder/bin/splunk display app
introspection_generator_addon CONFIGURED ENABLED INVISIBLE
[root@sosdev-idx3 apps]# /opt/fwd5/splunkforwarder/bin/splunk cmd btool server list introspection:generator:resource_usage --debug
/opt/fwd5/splunkforwarder/etc/apps/introspection_generator_addon/default/server.conf [introspection:generator:resource_usage]
/opt/fwd5/splunkforwarder/etc/apps/introspection_generator_addon/default/server.conf acquireExtra_i_data = false
/opt/fwd5/splunkforwarder/etc/apps/introspection_generator_addon/default/server.conf disabled = false
Note that now that the introspection_generator_addon app has been enabled by our app.conf push, the resource_usage data input is enabled! And indeed, the collector is running:
[root@sosdev-idx3 apps]# ps -ef | grep instrument-resource-usage
root 27697 27229 0 23:23 ? 00:00:00 /opt/fwd5/splunkforwarder/bin/splunkd instrument-resource-usage -p 8100
....and indeed, we get data:
[root@sosdev-idx3 apps]# tail -1 /opt/fwd5/splunkforwarder/var/log/introspection/resource_usage.log
{"datetime":"12-05-2014 23:33:06.435 -0800","log_level":"INFO","component":"Hostwide","data":{"mem":"7872.797","mem_used":"7100.086","swap":"14095.992","swap_used":"230.082","pg_paged_out":"8409558722","pg_swapped_out":"855499","forks":"86227439","runnable_process_count":"3","normalized_load_avg_1min":"0.91","cpu_user_pct":"66.43","cpu_system_pct":"5.97","cpu_idle_pct":"27.59"}}
... View more
