Splunk Search

How to collect DEBUG search artifacts (specifically for the search process) for Splunk Support?

splunkIT
Splunk Employee
Splunk Employee

For troubleshooting search related issue, Support would generally requests for debug search artifacts from the search head and, if for distributed search deployment, from at least one of the indexers (aka. search peers) for analysis. What are the steps for collecting these debug artifacts?

Tags (1)
0 Karma
1 Solution

splunkIT
Splunk Employee
Splunk Employee

Generally, Support would asks for debug artifacts from two scenarios for analysis and comparison:
- non-working scenario
- working scenario

(1) Prerequisite: enable search process DEBUG on search head and on at least one indexer (aka search peer):
On both the search head (ie. SH1.splunk.com-8089) and indexer (ie. IDX1.splunk-8081), edit $SPLUNK_HOME/etc/log-searchprocess.cfg
and change the following properties to enable debug logging and to increase the search.log file archives:

From:
rootCategory=INFO,searchprocessAppender
appender.searchprocessAppender.maxBackupIndex=3

To:
rootCategory=DEBUG,searchprocessAppender
appender.searchprocessAppender.maxBackupIndex=50

Save changes

Note 1: This change is global; affects all searches going forward. We strongly recommend reverting this change once the debug artifacts have been collected.
Note 2: $SPLUNK_HOME/etc/log-searchprocess.cfg file might be read-only by default. Please give proper file permission if needed.

(2) Run your search that would reproduce the behavior and gather search artifacts from both search head and the indexer/s (where you already have DEBUG search process enabled - see step 1).

(3) Get the search ID (SID) and create .tgz or .zip archives of search artifacts pertaining to SID in search head and indexer (under $SPLUNK_HOE/var/run/splunk/distpatch). For example, for SID 1494888652.9427, you should see the following search artifacts directories in $SPLUNK_HOME/var/run/splunk/distpatch:

On search head: $SPLUNK_HOE/var/run/splunk/distpatch/1494888652.9427
On indexer: $SPLUNK_HOE/var/run/splunk/distpatch/remote_SH1.splunk.com-8089_1494888652.9427

Note: see https://docs.splunk.com/Documentation/Splunk/latest/Search/ViewsearchjobpropertieswiththeJobInspecto... for details on obtaining the SID.

(4) Run your search for the expected behavior and gather search artifacts

(5) Repeat step 3 above to gather search artifacts
(6) Also collect diags of the search head and indexer used in the reproduction
(7) Attach the debug search artifacts (1 set for non-working scenario; 1 set for working scenario) to the support ticket.
(8) Attach diag of Search head and indexer to the support ticket

View solution in original post

0 Karma

splunkIT
Splunk Employee
Splunk Employee

Generally, Support would asks for debug artifacts from two scenarios for analysis and comparison:
- non-working scenario
- working scenario

(1) Prerequisite: enable search process DEBUG on search head and on at least one indexer (aka search peer):
On both the search head (ie. SH1.splunk.com-8089) and indexer (ie. IDX1.splunk-8081), edit $SPLUNK_HOME/etc/log-searchprocess.cfg
and change the following properties to enable debug logging and to increase the search.log file archives:

From:
rootCategory=INFO,searchprocessAppender
appender.searchprocessAppender.maxBackupIndex=3

To:
rootCategory=DEBUG,searchprocessAppender
appender.searchprocessAppender.maxBackupIndex=50

Save changes

Note 1: This change is global; affects all searches going forward. We strongly recommend reverting this change once the debug artifacts have been collected.
Note 2: $SPLUNK_HOME/etc/log-searchprocess.cfg file might be read-only by default. Please give proper file permission if needed.

(2) Run your search that would reproduce the behavior and gather search artifacts from both search head and the indexer/s (where you already have DEBUG search process enabled - see step 1).

(3) Get the search ID (SID) and create .tgz or .zip archives of search artifacts pertaining to SID in search head and indexer (under $SPLUNK_HOE/var/run/splunk/distpatch). For example, for SID 1494888652.9427, you should see the following search artifacts directories in $SPLUNK_HOME/var/run/splunk/distpatch:

On search head: $SPLUNK_HOE/var/run/splunk/distpatch/1494888652.9427
On indexer: $SPLUNK_HOE/var/run/splunk/distpatch/remote_SH1.splunk.com-8089_1494888652.9427

Note: see https://docs.splunk.com/Documentation/Splunk/latest/Search/ViewsearchjobpropertieswiththeJobInspecto... for details on obtaining the SID.

(4) Run your search for the expected behavior and gather search artifacts

(5) Repeat step 3 above to gather search artifacts
(6) Also collect diags of the search head and indexer used in the reproduction
(7) Attach the debug search artifacts (1 set for non-working scenario; 1 set for working scenario) to the support ticket.
(8) Attach diag of Search head and indexer to the support ticket

0 Karma

nickhills
Ultra Champion

Take a look at this article and pay attention to the warnings:

https://docs.splunk.com/Documentation/Splunk/7.2.4/Troubleshooting/Enabledebuglogging

You can generate diag files from the UI, under settings->instrumentation->new diag.
It might be worth asking support exactly which debug handlers they want turned on.

If my comment helps, please give it a thumbs up!
0 Karma

splunkIT
Splunk Employee
Splunk Employee

This method would require splunk to restart and might not be practical for production environment.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...