Monitoring Splunk

Unable to locate an inputs stanza sending a data to a particular index using btool ?

Hemnaath
Motivator

Hi All, we are removing the unwanted indexes from prod environment as they are deprecated, while performing this activity, we got stuck in finding the exact inputs.conf stanza monitoring and sending the data to a particular index, I had used below steps to find out the inputs stanza to disable and remove it from the host, but unable to get the index details in inputs.conf stanza.

steps:
1) From search head, we have got the host, source and sourcetype details by searching index=summary_forwarders.
2) Next logged into the particular indexer host from where the data are being sent to the index=summary_forwardes and executed the btool command "./splunk cmd btool --debug inputs list | grep summary_forwarders | more but unable to locate the stanza.
3) Tired to access the indexer instance via GUI but unable to access the web URL.

Kindly let me know is there any other method where we can filter out index detail from inputs.conf stanza details.

Tags (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

An index name like "summary_forwarders" sounds like a summary index. If that's what it is then you won't (shouldn't) find any references to it in inputs.conf. Summary indexes are written to by scheduled searches. Look in your savedsearches.conf files for references to that index.

find $SPLUNK_HOME -name savedsearches.conf -print0 | xargs -r0 grep summary_forwarders
---
If this reply helps you, Karma would be appreciated.
0 Karma

Hemnaath
Motivator

Hi richgalloway, thanks for your effort on this, but when tried the above find command, it did not fetch any output.

Need to find for all the below list of indexes details
summary_forwarders
summary_indexers
summary_sourcetypes
summary_sources
summary_hosts
summary_pools

Kindly guide me, is there any other way i can narrow down the inputs.conf details for the above list of indexes details.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

For each index, search for the sources that write to that index. Then search inputs.conf for those sources.

| metadata type=sources index=summary_forwarders

If you use universal forwarders then you will need to search in the inputs.conf files on the forwarders (unless you have local copies).

---
If this reply helps you, Karma would be appreciated.
0 Karma

Hemnaath
Motivator

Hi richgalloway, I had tried above search and found the source type and source, then ran the btool to find out the location of the inputs.conf stanza in the indexer instance host but no luck.

index=summary_forwarders source="All forwarders - regenerator summary index" sourcetype=stash

btool details :
./splunk cmd btool inputs list --debug | grep All forwarders | more
grep: forwarders: No such file or directory
so decided to run the btool against the sourcetype=stash and found some output but there was no source mapped to it.
btool details:
./splunk cmd btool inputs list --debug | grep stash | more
/opt/splunk/etc/system/default/inputs.conf

[batch:///opt/splunk/var/spool/splunk/...stash_new]
/opt/splunk/etc/system/default/inputs.conf queue = stashparsing
/opt/splunk/etc/system/default/inputs.conf sourcetype = stash_new

Executed the btool against the props.conf file to find out the location, but no luck.
./splunk cmd btool props list --debug | grep stash | more
/opt/splunk/etc/system/default/props.conf [source::...stash]
/opt/splunk/etc/system/default/props.conf sourcetype = stash
/opt/splunk/etc/system/default/props.conf [source::...stash_new]
/opt/splunk/etc/system/default/props.conf sourcetype = stash_new
/opt/splunk/etc/system/default/props.conf [stash]
/opt/splunk/etc/system/default/props.conf REPORT-1 = stash_extract
/opt/splunk/etc/system/default/props.conf [stash_new]
/opt/splunk/etc/system/default/props.conf TRANSFORMS-sourcetype = set_sourcetype_to_stash

kindly guide how to get this fixed, we need to remove above mentioned index details from indexes.conf before removing we want to stop these index to ingest data into it.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Like I suspected originally, you're working with summary indexes. This is shown by "sourcetype=stash". That means you will not find references to the index in inputs.conf. Look in savedsearches.conf.

---
If this reply helps you, Karma would be appreciated.
0 Karma

Hemnaath
Motivator

HI richgalloway, you're right, i could see the config details for all the indexes which are listed in the above comments in savedsearches.conf, so by disabling the index details in savedsearches.conf will stop sending the data to these indexes, is that correct. And also please guide us why it is configured like this usually monitoring details will be in inputs.conf right.

Btool details:

./splunk cmd btool savedsearches list --debug | grep summary_forwarders | more
/opt/splunk/etc/apps/SplunkDeploymentMonitor/default/savedsearches.conf search = index="summary_forwarders" | delete
/opt/splunk/etc/apps/SplunkDeploymentMonitor/default/savedsearches.conf action.summary_index._name = summary_forwarders
/opt/splunk/etc/apps/SplunkDeploymentMonitor/default/savedsearches.conf search = index="summary_forwarders" | eval mb=kb/1024 | eval _time = _time+1800 | timechart pa
rtial=f sum(mb) as MB by sourceHost

thanks in advance.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

You have a saved search, defined in "action.summary_index._name" line, that puts its results into a summary index called "summary_forwarders". You have a two options.
1) Disable this search so it doesn't run and doesn't write to the index.
2) Change the index name so it writes to a different index.

You also have a couple of searches that are reading data from the 'summary_forwarders' index. These searches must be disabled or modified before the index can be deleted.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...