Why are the indexers trying to execute these command if they are defined as 'local = true'?

[wegscd]

We've had some custom commands defined on our indexers for years. Here is /opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf:

[netbotzreport]
filename = netbotzreport.py
enableheader = true
outputheader = true
requires_srinfo = true
stderr_dest = message
supports_getinfo = true
supports_rawargs = true
supports_multivalues = true
local = true
streaming = true
# this should not be necessary
overrides_timeorder = true
required_fields=mib,oid,snmp_index,value

[netbotzextract]
filename = netbotzextract.py
enableheader = true
outputheader = true
requires_srinfo = true
stderr_dest = message
supports_getinfo = true
supports_rawargs = true
supports_multivalues = true
local = true
streaming = true
# this should not be necessary
overrides_timeorder = true

[pipesniff]
filename = pipesniff.py
enableheader = true
outputheader = true
requires_srinfo = true
stderr_dest = message
supports_getinfo = true
supports_rawargs = true
supports_multivalues = true
local = true
streaming = true
Sometime in the last month, searches using these commands have started failing with these messages from the indexers:

[awnulsplunkp1] Search Factory: Unknown search command 'netbotzextract'.

We did a 6.5 -> 7.0 last week, which I suspect is what changed.

Why are the indexers trying to execute these command if they are defined as 'local = true'?

[deepashri_123]

Hey wegscd,

Any customization that is done has to be done in /opt/splunk/etc/apps/whirlpool_netbotz/local/commands.conf that is local and not in default directory.
The changes that were done in default directory got overwritten after the upgrade.

Create a commands.conf file in local directory in your app and add the changes there.
And you can cross check what configs are used by indexer by running following command on indexer

/$SPLUNK_HOME$/bin/splunk cmd btool commands list --debug

[wegscd]

there is nothing in local/ to override default/commands,conf, and nothing there got overwritten in the upgrade. The btool says that the local = true in default is being used.

/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         [netbotzextract]
/opt/splunk/etc/system/default/commands.conf                         changes_colorder = true
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         enableheader = true
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         filename = netbotzextract.py
/opt/splunk/etc/system/default/commands.conf                         generates_timeorder = false
/opt/splunk/etc/system/default/commands.conf                         generating = false
/opt/splunk/etc/system/default/commands.conf                         is_risky = false
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         local = true
/opt/splunk/etc/system/default/commands.conf                         maxinputs = 50000
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         outputheader = true
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         overrides_timeorder = true
/opt/splunk/etc/system/default/commands.conf                         passauth = false
/opt/splunk/etc/system/default/commands.conf                         perf_warn_limit = 0
/opt/splunk/etc/system/default/commands.conf                         required_fields = *
/opt/splunk/etc/system/default/commands.conf                         requires_preop = false
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         requires_srinfo = true
/opt/splunk/etc/system/default/commands.conf                         retainsevents = false
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         stderr_dest = message
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         streaming = true
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         supports_getinfo = true
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         supports_multivalues = true
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         supports_rawargs = true
/opt/splunk/etc/system/default/commands.conf                         type = python
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         [netbotzreport]
/opt/splunk/etc/system/default/commands.conf                         changes_colorder = true
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         enableheader = true
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         filename = netbotzreport.py
/opt/splunk/etc/system/default/commands.conf                         generates_timeorder = false
/opt/splunk/etc/system/default/commands.conf                         generating = false
/opt/splunk/etc/system/default/commands.conf                         is_risky = false
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         local = true
/opt/splunk/etc/system/default/commands.conf                         maxinputs = 50000
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         outputheader = true
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         overrides_timeorder = true
/opt/splunk/etc/system/default/commands.conf                         passauth = false
/opt/splunk/etc/system/default/commands.conf                         perf_warn_limit = 0
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         required_fields = mib,oid,snmp_index,value
/opt/splunk/etc/system/default/commands.conf                         requires_preop = false
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         requires_srinfo = true
/opt/splunk/etc/system/default/commands.conf                         retainsevents = false
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         stderr_dest = message
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         streaming = true
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         supports_getinfo = true
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         supports_multivalues = true
/opt/splunk/etc/apps/whirlpool_netbotz/default/commands.conf         supports_rawargs = true
/opt/splunk/etc/system/default/commands.conf                         type = python

[kiril123]

I am having the same problem.