How to collect DEBUG search artifacts (specifically for the search process) for Splunk Support?

[splunkIT]

For troubleshooting search related issue, Support would generally requests for debug search artifacts from the search head and, if for distributed search deployment, from at least one of the indexers (aka. search peers) for analysis. What are the steps for collecting these debug artifacts?

[splunkIT · Accepted solution]

Generally, Support would asks for debug artifacts from two scenarios for analysis and comparison:
- non-working scenario
- working scenario

(1) Prerequisite: enable search process DEBUG on search head and on at least one indexer (aka search peer):
On both the search head (ie. SH1.splunk.com-8089) and indexer (ie. IDX1.splunk-8081), edit $SPLUNK_HOME/etc/log-searchprocess.cfg
and change the following properties to enable debug logging and to increase the search.log file archives:

From:
rootCategory=INFO,searchprocessAppender
appender.searchprocessAppender.maxBackupIndex=3

To:
rootCategory=DEBUG,searchprocessAppender
appender.searchprocessAppender.maxBackupIndex=50

Save changes

Note 1: This change is global; affects all searches going forward. We strongly recommend reverting this change once the debug artifacts have been collected.
Note 2: $SPLUNK_HOME/etc/log-searchprocess.cfg file might be read-only by default. Please give proper file permission if needed.

(2) Run your search that would reproduce the behavior and gather search artifacts from both search head and the indexer/s (where you already have DEBUG search process enabled - see step 1).

(3) Get the search ID (SID) and create .tgz or .zip archives of search artifacts pertaining to SID in search head and indexer (under $SPLUNK_HOE/var/run/splunk/distpatch). For example, for SID 1494888652.9427, you should see the following search artifacts directories in $SPLUNK_HOME/var/run/splunk/distpatch:

On search head: $SPLUNK_HOE/var/run/splunk/distpatch/1494888652.9427
On indexer: $SPLUNK_HOE/var/run/splunk/distpatch/remote_SH1.splunk.com-8089_1494888652.9427

Note: see https://docs.splunk.com/Documentation/Splunk/latest/Search/ViewsearchjobpropertieswiththeJobInspecto... for details on obtaining the SID.

(4) Run your search for the expected behavior and gather search artifacts

(5) Repeat step 3 above to gather search artifacts
(6) Also collect diags of the search head and indexer used in the reproduction
(7) Attach the debug search artifacts (1 set for non-working scenario; 1 set for working scenario) to the support ticket.
(8) Attach diag of Search head and indexer to the support ticket

[nickhills]

Take a look at this article and pay attention to the warnings:

https://docs.splunk.com/Documentation/Splunk/7.2.4/Troubleshooting/Enabledebuglogging

You can generate diag files from the UI, under settings->instrumentation->new diag.
It might be worth asking support exactly which debug handlers they want turned on.

[splunkIT]

This method would require splunk to restart and might not be practical for production environment.