Deployment Architecture

manual splunk-optimize

yannK
Splunk Employee
Splunk Employee

I noticed that I have some buckets with many tsidx files. I know that the hot buckets are being optimized on a regular basis by splunk. But I would like to optimize manually the warm and cold buckets.

see http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Optimizeindexes

yannK
Splunk Employee
Splunk Employee

you can run splunk optimize manually with

splunk-optimize bucket_folder

usage:./splunk-optimize -d|--directory <dir>
[-h|--help]
[-m|--mode <normal|force|all> (defaults to normal)]
[-v|--verbose]
[-s|--min-src-count <number> (defaults to 8)]
[-i|--iterations-max <number> (defaults to unlmitted)]
[-b|--lex-tpb <number> (default 64 - merged lexicon terms per block)]
[-x|--max-allowed-size <number> (max allowed extra disk space required, in bytes)]
[-p]--page-size <bytes> (memory allocation size, default: 1048576 (1MB), minimum: 16384)]


and you can retrieve the list of all the buckets with a lot of tsidx files with this script
`


#!/bin/bash
# find_stidx.sh script for listing the buckets with too many tsidx
# required the base folder as argument

#settings
tsidx_limit=10
verbose=1 # display the count per folder
#verbose=0 # just display list of folders
include_hot=1 # look in hot and warm and cold buckets

if [ $# -lt 1 ]; then
echo 1>&2 "usage : $0 "
exit 2
fi

hot_bucket_list=""
db_bucket_list=""
base_folder=$1

# get the list of the buckets folders
if [ $include_hot -eq 1 ] ; then
hot_bucket_list=find $base_folder -name "hot_*"
fi
bucket_list=find $base_folder -name "db_*"
bucket_list="$hot_nucket_list $bucket_list"

# count the tsidx
if [ $verbose -eq 1 ] ; then
echo "list of buckets with more than $tsidx_limit tsidx files"

fi
for bucket in $bucket_list ; do
count=find $bucket -name "*.tsidx" | wc -l
if [ "$count" -gt "$tsidx_limit" ] ; then
if [ $verbose -eq 1 ] ; then
echo "$count tsidx in $bucket"
else
echo "$bucket"
fi
fi
done
`

khyoung7410
Communicator

Thank you
How to use script?

0 Karma

kristian_kolb
Ultra Champion

$SPLUNK_HOME/bin/splunk-optimize -d /h_data/splunk/splunk/var/lib/splunk/idx_3/db/<your bucket with many tsidx files here>

khyoung7410
Communicator

my index name is idx_3
directory is
/h_data/splunk/splunk/var/lib/splunk/idx_3/db
How to use?

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...