Hi All, we are removing the unwanted indexes from prod environment as they are deprecated, while performing this activity, we got stuck in finding the exact inputs.conf stanza monitoring and sending the data to a particular index, I had used below steps to find out the inputs stanza to disable and remove it from the host, but unable to get the index details in inputs.conf stanza.
1) From search head, we have got the host, source and sourcetype details by searching index=summary_forwarders.
2) Next logged into the particular indexer host from where the data are being sent to the index=summary_forwardes and executed the btool command "./splunk cmd btool --debug inputs list | grep summary_forwarders | more but unable to locate the stanza.
3) Tired to access the indexer instance via GUI but unable to access the web URL.
Kindly let me know is there any other method where we can filter out index detail from inputs.conf stanza details.
An index name like "summary_forwarders" sounds like a summary index. If that's what it is then you won't (shouldn't) find any references to it in inputs.conf. Summary indexes are written to by scheduled searches. Look in your savedsearches.conf files for references to that index.
find $SPLUNK_HOME -name savedsearches.conf -print0 | xargs -r0 grep summary_forwarders
Hi richgalloway, thanks for your effort on this, but when tried the above find command, it did not fetch any output.
Need to find for all the below list of indexes details
Kindly guide me, is there any other way i can narrow down the inputs.conf details for the above list of indexes details.
For each index, search for the sources that write to that index. Then search inputs.conf for those sources.
| metadata type=sources index=summary_forwarders
If you use universal forwarders then you will need to search in the inputs.conf files on the forwarders (unless you have local copies).
Hi richgalloway, I had tried above search and found the source type and source, then ran the btool to find out the location of the inputs.conf stanza in the indexer instance host but no luck.
index=summary_forwarders source="All forwarders - regenerator summary index" sourcetype=stash
btool details :
./splunk cmd btool inputs list --debug | grep All forwarders | more
grep: forwarders: No such file or directory
so decided to run the btool against the sourcetype=stash and found some output but there was no source mapped to it.
./splunk cmd btool inputs list --debug | grep stash | more
/opt/splunk/etc/system/default/inputs.conf queue = stashparsing
/opt/splunk/etc/system/default/inputs.conf sourcetype = stash_new
Executed the btool against the props.conf file to find out the location, but no luck.
./splunk cmd btool props list --debug | grep stash | more
/opt/splunk/etc/system/default/props.conf sourcetype = stash
/opt/splunk/etc/system/default/props.conf sourcetype = stash_new
/opt/splunk/etc/system/default/props.conf REPORT-1 = stash_extract
/opt/splunk/etc/system/default/props.conf TRANSFORMS-sourcetype = set_sourcetype_to_stash
kindly guide how to get this fixed, we need to remove above mentioned index details from indexes.conf before removing we want to stop these index to ingest data into it.
Like I suspected originally, you're working with summary indexes. This is shown by "sourcetype=stash". That means you will not find references to the index in inputs.conf. Look in savedsearches.conf.
HI richgalloway, you're right, i could see the config details for all the indexes which are listed in the above comments in savedsearches.conf, so by disabling the index details in savedsearches.conf will stop sending the data to these indexes, is that correct. And also please guide us why it is configured like this usually monitoring details will be in inputs.conf right.
./splunk cmd btool savedsearches list --debug | grep summary_forwarders | more
/opt/splunk/etc/apps/SplunkDeploymentMonitor/default/savedsearches.conf search = index="summary_forwarders" | delete
/opt/splunk/etc/apps/SplunkDeploymentMonitor/default/savedsearches.conf action.summary_index._name = summary_forwarders
/opt/splunk/etc/apps/SplunkDeploymentMonitor/default/savedsearches.conf search = index="summary_forwarders" | eval mb=kb/1024 | eval _time = _time+1800 | timechart pa
rtial=f sum(mb) as MB by sourceHost
thanks in advance.
You have a saved search, defined in "action.summary_index._name" line, that puts its results into a summary index called "summary_forwarders". You have a two options.
1) Disable this search so it doesn't run and doesn't write to the index.
2) Change the index name so it writes to a different index.
You also have a couple of searches that are reading data from the 'summary_forwarders' index. These searches must be disabled or modified before the index can be deleted.