Splunk App for CEF: How to resolve error "Search Factory: Unknown search command 'cefout'" from indexer cluster peers?

[mantod]

I've installed and configured the Splunk App for CEF 2.0.0 on Splunk Enterprise 6.6.0. I've created a single CEF output and installed the generated cefout add-on to each indexer. It works fine for standalone indexers, but fails on each indexer cluster peer with the error (remote_searches.log):

05-22-2017 10:02:27.446 +0000 ERROR StreamedSearch - sid=remote_ip-{SEARCH HEAD}_rt_scheduler__admin_c3BsdW5rX2FwcF9jZWY__RMD5b4adc662619c6e71_at_1495447345_6, Search Factory: Unknown search command 'cefout'.

I can see the indexers have the command replicated from the search head:

/opt/splunk/var/run/searchpeers/ip-{SEARCH HEAD}-1495445826/apps/splunk_app_cef/bin/cefout.py

I don't understand why they're not using it, given that the non-clustered indexers use the same just fine. What am I missing?

(FYI, I've worked around this problem for now by manually adding the cefout command to the generated cefout bundle. But I want to get to the point where I can use the generated bundle without manual changes, to avoid the chance of user error as administrators make further changes).

Any advice would be greatly appreciated. Thanks.

[hazekamp]

Be advised that the cefout search command and corresponding commands.conf need not be distributed to the indexer tier. The cefout command and corresponding commands.conf should be distributed to the indexer tier automatically via distributed search bundle replication. The more likely issue here is that something with distributed search bundle replication is not behaving properly.

We're not 100% certain of the root cause at this juncture, but there is at least one report that setting an explicit whitelist for splunk_app_cef files in distsearch.conf can mitigate the issue:

[replicationWhitelist] 
cef = apps[/\\]splunk_app_cef[/\\]...

Update: There's also reports that the error is occurring even when artifacts are properly replicated (and we have a working reproduction of this issue). There is a bug open with Splunk Enterprise pertaining to custom streaming commands not correctly being acquired from the bundle. It would make sense that placing the bin directory and corresponding commands.conf in an app installed directly on the indexer would help mitigate this specific issue.

Update: cefout.py refers to other libraries, so best to just include bin dir

[DavidH1]

I had this exact issue, but I am on a clustered search head and clustered indexer environment. I fixed this by moving the splunk_app_cef/bin folder and the splunk_app_cef/default/commands.conf to the Splunk_TA_cefout app on the indexers and it resolved my issue.