Deployment Architecture

manual splunk-optimize

yannK
Splunk Employee
Splunk Employee

I noticed that I have some buckets with many tsidx files. I know that the hot buckets are being optimized on a regular basis by splunk. But I would like to optimize manually the warm and cold buckets.

see http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Optimizeindexes

yannK
Splunk Employee
Splunk Employee

you can run splunk optimize manually with

splunk-optimize bucket_folder

usage:./splunk-optimize -d|--directory <dir>
[-h|--help]
[-m|--mode <normal|force|all> (defaults to normal)]
[-v|--verbose]
[-s|--min-src-count <number> (defaults to 8)]
[-i|--iterations-max <number> (defaults to unlmitted)]
[-b|--lex-tpb <number> (default 64 - merged lexicon terms per block)]
[-x|--max-allowed-size <number> (max allowed extra disk space required, in bytes)]
[-p]--page-size <bytes> (memory allocation size, default: 1048576 (1MB), minimum: 16384)]


and you can retrieve the list of all the buckets with a lot of tsidx files with this script
`


#!/bin/bash
# find_stidx.sh script for listing the buckets with too many tsidx
# required the base folder as argument

#settings
tsidx_limit=10
verbose=1 # display the count per folder
#verbose=0 # just display list of folders
include_hot=1 # look in hot and warm and cold buckets

if [ $# -lt 1 ]; then
echo 1>&2 "usage : $0 "
exit 2
fi

hot_bucket_list=""
db_bucket_list=""
base_folder=$1

# get the list of the buckets folders
if [ $include_hot -eq 1 ] ; then
hot_bucket_list=find $base_folder -name "hot_*"
fi
bucket_list=find $base_folder -name "db_*"
bucket_list="$hot_nucket_list $bucket_list"

# count the tsidx
if [ $verbose -eq 1 ] ; then
echo "list of buckets with more than $tsidx_limit tsidx files"

fi
for bucket in $bucket_list ; do
count=find $bucket -name "*.tsidx" | wc -l
if [ "$count" -gt "$tsidx_limit" ] ; then
if [ $verbose -eq 1 ] ; then
echo "$count tsidx in $bucket"
else
echo "$bucket"
fi
fi
done
`

khyoung7410
Communicator

Thank you
How to use script?

0 Karma

kristian_kolb
Ultra Champion

$SPLUNK_HOME/bin/splunk-optimize -d /h_data/splunk/splunk/var/lib/splunk/idx_3/db/<your bucket with many tsidx files here>

khyoung7410
Communicator

my index name is idx_3
directory is
/h_data/splunk/splunk/var/lib/splunk/idx_3/db
How to use?

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...