topic Re: Please route me i am at INDEX deadend. in Getting Data In

Please route me i am at INDEX deadend.

Rocky31 — Tue, 04 Jun 2019 22:01:09 GMT

I appreciate your time and effort. below are questions

1) I want to find out where is the index.conf for my index XYZ (read bunch of docs not sure where to check in SH or Deployment server or something else)

2) with this search i got bunch of index (search “|tstats count where source=/opt/splunk/syslog-ng/* by index | table index”) where is index.conf configs for those indexes?

3) And where to check those indexes over 50% of max size?

Re: Please route me i am at INDEX deadend.

MuS — Tue, 04 Jun 2019 23:13:54 GMT

Hi Rocky31,

1 & 2 can be resolved by using this command on an indexer instances:

$SPLUNK_HOME/bin/splunk btool indexes list --debug

This will show you the output of all indexes.conf merged and their location. For just one index run it like this:

$SPLUNK_HOME/bin/splunk btool indexes list indexNameHere --debug

Regarding 3. if you have the monitoring console configured https://docs.splunk.com/Documentation/Splunk/latest/DMC/DMCoverview you will get a lot of pre-built dashboards around index sizing, age and more.

Hope this helps ...

cheers, MuS

Re: Please route me i am at INDEX deadend.

Rocky31 — Wed, 05 Jun 2019 15:11:32 GMT

thanks for your answer, can you also tell me how to check each index size occupied in indexers in percentages

Re: Please route me i am at INDEX deadend.

woodcock — Wed, 05 Jun 2019 16:26:52 GMT

Interestingly enough, I just wrote this code last week:

#! /bin/bash

SPLUNK_DB=$(/opt/splunk/bin/splunk envvars | sed 's/.*SPLUNK_DB=\(\S\+\).*$/\1/')
if [[ $(echo ${SPLUNK_DB} | awk -F '/' '{print NF}') -eq 0 ]]; then
   # No '/' characters so splunk is not running here
   TEST_MODE="YES"
   echo "TEST MODE INITIATED"
   SPLUNK_DB="/opt/splunk/var/lib/splunk"
   echo "SPLUNK_DB=${SPLUNK_DB}"
   LOGFILE="/dev/null"
   echo "LOGFILE=${LOGFILE}"
   indexes_file=./indexes.conf
   # assume that the index detail file already exists locally
else # Splunk is running here; assume that it really is an indexer
   LOGFILE="/var/log/$0.$(date +%Y%m%d_%H%M%S)"
   indexes_file=/tmp/indexes.conf
   # generate the index detail file
   /opt/splunk/bin/splunk btool indexes list | egrep '^\[|^homePath\s*=|^path\s*=' > ${indexes_file}
fi

while read -r line; do
   if [ "$(echo ${line} | cut -c 1)" != '[' ] || [ "$(echo ${line} | cut -c1-8)" = '[volume:' ] || [ "$(echo ${line} | cut -c1-17)" = '[provider-family:' ]; then
      continue
   fi
   # capture next index value
   index=$(echo ${line} | awk '{print substr($0, 2, length($0) - 2)}')
   index_lower=$(echo ${index} | tr "[:upper:]" "[:lower:]")
   echo "   index=${index}, index_lower=${index_lower}"
   if [[ $index == "default" ]] || [[ $index == "splunklogger" ]] || [[ $index == "history" ]]; then
      echo "Skipping this one..."
      continue # skip this one
   fi
   # Identifying warm buckets path
   home_path=$(grep -A1 -w "^\[${index}" ${indexes_file} | grep -E "^homePath\s*=" | awk -F '=' '{print $2}'| xargs echo)
   echo "home_path=${home_path}"
   if [[ $home_path == volume:* ]]; then # dereference volume
      warm_volume_name=${home_path%%/*}
      # Identifying Warm Volume Path
      warm_volume_path=$(grep -A1 "^\[${warm_volume_name}" ${indexes_file} | grep path | awk -F '=' '{print $2 }' | xargs echo)
      echo "warm_volume_path=${warm_volume_path}"
   elif [[ $home_path == \$SPLUNK_DB* ]]; then # dereference "$SPLUNK_DB"
      warm_volume_path=${SPLUNK_DB}
   else # just a normal path that needs no modification at all
      warm_volume_path=''
   fi
   home_path=${home_path#*/}
   comp_path=$(echo ${warm_volume_path}/${home_path} | tr "[:upper:]" "[:lower:]")
   # replace "$_index_name" if present
   comp_path=$(echo ${comp_path} | sed "s%/\$_index_name/%/${index_lower}/%")
   echo "comp_path=${comp_path}"

   if [[ ${TEST_MODE} == "YES" ]]; then
      # Not running on an actual indexer (splunk is not running here)
      continue
   fi

   # Do other stuff here
done < ${indexes_file}

Re: Please route me i am at INDEX deadend.

Rocky31 — Thu, 06 Jun 2019 18:49:17 GMT

Hi Splunk superstar,
this is really a out of box thinking, i need to work on it to understand further. good one.

Re: Please route me i am at INDEX deadend.

woodcock — Sun, 09 Jun 2019 21:24:08 GMT

This answer show which configuration files contain the definitions for every index, but it does not fully resolve the path, which is what I believe the OP desires. Please see my answer for how to do that.

Re: Please route me i am at INDEX deadend.

MuS — Sun, 09 Jun 2019 22:01:41 GMT

You must be using a different version of btool 😉
I get a nice output of the path when using the posted command $SPLUNK_HOME/bin/splunk btool indexes list --debug

cheers, MuS

Re: Please route me i am at INDEX deadend.

Rocky31 — Tue, 11 Jun 2019 16:29:14 GMT

Sure, thanks

Re: Please route me i am at INDEX deadend.

woodcock — Tue, 11 Jun 2019 16:59:11 GMT

You get a value for the path but it will not resolve any volume definitions, nor $SPLUNK_DB, which is what my code does.

Re: Please route me i am at INDEX deadend.

woodcock — Sun, 14 Jul 2019 18:29:51 GMT

Be sure to come back after you test and click Accept to close the question (or to point out bugs/fixes)!

Re: Please route me i am at INDEX deadend.

woodcock — Thu, 26 Dec 2019 01:44:00 GMT

So how did it work out @Rocky31?