Splunk Enterprise

How to generate a report on multiple indexes?

Brian_Osburn
Builder

There's a limitation in the dbinspect command where you cannot specify multiple indexes to report on, therefore reporting on an installation where multiple indexes are used can be a time consuming thing.

This answers article is a good start, but unfortunately you can only do one index at a time. How do I get around this?

Tags (2)
1 Solution

Brian_Osburn
Builder

This PERL script will generate a csv report and email it (assuming all required packages are installed) to specified email address.

Only thing required is the Shared utilities package (in my case it was sharutils-4.6.1-2.x86_64) for the uuencode portion.

You will need to modify some of the variables (mainly the $username and $password) if you want it to automatically log you in.

!/usr/bin/perl

### Set variables
$splunk_bin_dir="/opt/splunk/bin";
$mail_to="$ARGV[0]";
$header="Index Name,DB Type,earliest event time,latest event time, size (mb)";
$output_dir="/tmp";
$report_name="index_report.csv";
$username="admin";
$password="passwordhere!";

### Delete previous versions of the report

$output_name="> ${output_dir}/${report_name}";

open(OUTPUT,$output_name);


### Get list of indexes
@index_config_raw=`cat /opt/splunk/etc/system/local/indexes.conf`;

foreach $line (@index_config_raw) {
        chomp $line;

        if ($line=~m/\[/) {
                $line=~m/\[(.*)\]/g;
                $raw_index=$1;

                push(@indexes,$raw_index);

        }
}

print OUTPUT "$header \n";

### Processes indexes
foreach $index (@indexes) {
        chomp $index;

        $splunk_command="${splunk_bin_dir}/splunk search \"| dbinspect index=\"${index}\" timeformat=\"\%s\" | rename state as category | stats min(earliestTime) as earliestTime max(latestTime) as latestTime sum(sizeOnDiskMB) as MB by category | convert timeformat=\"\%m/\%d/\%Y\" ctime(earliestTime) as earliestTime ctime(latestTime) as latestTime\" -auth ${username}:${password}| grep -v \"category\" | grep -v \"-\" ";


        @result=`${splunk_command}`;

        if ($#result ne "-1") {
                foreach $return (@result) {
                        chomp $return;

                        $return=~m/(hot|warm|cold|frozen)\s+([\d]+\/[\d]+\/[\d]+)\s+([\d]+\/[\d]+\/[\d]+)\s+([\d]+\.[\d]+)/gi;

                        $db_type=$1;
                        $earliest_event=$2;
                        $latest_event=$3;
                        $size=$4;

                        print OUTPUT "$index,$db_type,$earliest_event,$latest_event,$size \n";


                }
        }

        if ($#result eq "-1") {
                print OUTPUT "$index,Empty Index,Empty Index,Empty Index \n";
        }


}

if ($mail_to ne "") {
        `uuencode ${output_dir}/${report_name} ${output_dir}/${report_name} | mailx -s \"Splunk Index Report\" $mail_to`;
}

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

I will also note that dbinspect does not work in distributed search mode, it only does the local server. If you have a few indexers, this can also be tedious. That would be a nice enhancement too. If someone were interested.

Brian_Osburn
Builder

This PERL script will generate a csv report and email it (assuming all required packages are installed) to specified email address.

Only thing required is the Shared utilities package (in my case it was sharutils-4.6.1-2.x86_64) for the uuencode portion.

You will need to modify some of the variables (mainly the $username and $password) if you want it to automatically log you in.

!/usr/bin/perl

### Set variables
$splunk_bin_dir="/opt/splunk/bin";
$mail_to="$ARGV[0]";
$header="Index Name,DB Type,earliest event time,latest event time, size (mb)";
$output_dir="/tmp";
$report_name="index_report.csv";
$username="admin";
$password="passwordhere!";

### Delete previous versions of the report

$output_name="> ${output_dir}/${report_name}";

open(OUTPUT,$output_name);


### Get list of indexes
@index_config_raw=`cat /opt/splunk/etc/system/local/indexes.conf`;

foreach $line (@index_config_raw) {
        chomp $line;

        if ($line=~m/\[/) {
                $line=~m/\[(.*)\]/g;
                $raw_index=$1;

                push(@indexes,$raw_index);

        }
}

print OUTPUT "$header \n";

### Processes indexes
foreach $index (@indexes) {
        chomp $index;

        $splunk_command="${splunk_bin_dir}/splunk search \"| dbinspect index=\"${index}\" timeformat=\"\%s\" | rename state as category | stats min(earliestTime) as earliestTime max(latestTime) as latestTime sum(sizeOnDiskMB) as MB by category | convert timeformat=\"\%m/\%d/\%Y\" ctime(earliestTime) as earliestTime ctime(latestTime) as latestTime\" -auth ${username}:${password}| grep -v \"category\" | grep -v \"-\" ";


        @result=`${splunk_command}`;

        if ($#result ne "-1") {
                foreach $return (@result) {
                        chomp $return;

                        $return=~m/(hot|warm|cold|frozen)\s+([\d]+\/[\d]+\/[\d]+)\s+([\d]+\/[\d]+\/[\d]+)\s+([\d]+\.[\d]+)/gi;

                        $db_type=$1;
                        $earliest_event=$2;
                        $latest_event=$3;
                        $size=$4;

                        print OUTPUT "$index,$db_type,$earliest_event,$latest_event,$size \n";


                }
        }

        if ($#result eq "-1") {
                print OUTPUT "$index,Empty Index,Empty Index,Empty Index \n";
        }


}

if ($mail_to ne "") {
        `uuencode ${output_dir}/${report_name} ${output_dir}/${report_name} | mailx -s \"Splunk Index Report\" $mail_to`;
}
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...