Splunk Enterprise

How to generate a report on multiple indexes?

Brian_Osburn
Builder

There's a limitation in the dbinspect command where you cannot specify multiple indexes to report on, therefore reporting on an installation where multiple indexes are used can be a time consuming thing.

This answers article is a good start, but unfortunately you can only do one index at a time. How do I get around this?

Tags (2)
1 Solution

Brian_Osburn
Builder

This PERL script will generate a csv report and email it (assuming all required packages are installed) to specified email address.

Only thing required is the Shared utilities package (in my case it was sharutils-4.6.1-2.x86_64) for the uuencode portion.

You will need to modify some of the variables (mainly the $username and $password) if you want it to automatically log you in.

!/usr/bin/perl

### Set variables
$splunk_bin_dir="/opt/splunk/bin";
$mail_to="$ARGV[0]";
$header="Index Name,DB Type,earliest event time,latest event time, size (mb)";
$output_dir="/tmp";
$report_name="index_report.csv";
$username="admin";
$password="passwordhere!";

### Delete previous versions of the report

$output_name="> ${output_dir}/${report_name}";

open(OUTPUT,$output_name);


### Get list of indexes
@index_config_raw=`cat /opt/splunk/etc/system/local/indexes.conf`;

foreach $line (@index_config_raw) {
        chomp $line;

        if ($line=~m/\[/) {
                $line=~m/\[(.*)\]/g;
                $raw_index=$1;

                push(@indexes,$raw_index);

        }
}

print OUTPUT "$header \n";

### Processes indexes
foreach $index (@indexes) {
        chomp $index;

        $splunk_command="${splunk_bin_dir}/splunk search \"| dbinspect index=\"${index}\" timeformat=\"\%s\" | rename state as category | stats min(earliestTime) as earliestTime max(latestTime) as latestTime sum(sizeOnDiskMB) as MB by category | convert timeformat=\"\%m/\%d/\%Y\" ctime(earliestTime) as earliestTime ctime(latestTime) as latestTime\" -auth ${username}:${password}| grep -v \"category\" | grep -v \"-\" ";


        @result=`${splunk_command}`;

        if ($#result ne "-1") {
                foreach $return (@result) {
                        chomp $return;

                        $return=~m/(hot|warm|cold|frozen)\s+([\d]+\/[\d]+\/[\d]+)\s+([\d]+\/[\d]+\/[\d]+)\s+([\d]+\.[\d]+)/gi;

                        $db_type=$1;
                        $earliest_event=$2;
                        $latest_event=$3;
                        $size=$4;

                        print OUTPUT "$index,$db_type,$earliest_event,$latest_event,$size \n";


                }
        }

        if ($#result eq "-1") {
                print OUTPUT "$index,Empty Index,Empty Index,Empty Index \n";
        }


}

if ($mail_to ne "") {
        `uuencode ${output_dir}/${report_name} ${output_dir}/${report_name} | mailx -s \"Splunk Index Report\" $mail_to`;
}

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

I will also note that dbinspect does not work in distributed search mode, it only does the local server. If you have a few indexers, this can also be tedious. That would be a nice enhancement too. If someone were interested.

Brian_Osburn
Builder

This PERL script will generate a csv report and email it (assuming all required packages are installed) to specified email address.

Only thing required is the Shared utilities package (in my case it was sharutils-4.6.1-2.x86_64) for the uuencode portion.

You will need to modify some of the variables (mainly the $username and $password) if you want it to automatically log you in.

!/usr/bin/perl

### Set variables
$splunk_bin_dir="/opt/splunk/bin";
$mail_to="$ARGV[0]";
$header="Index Name,DB Type,earliest event time,latest event time, size (mb)";
$output_dir="/tmp";
$report_name="index_report.csv";
$username="admin";
$password="passwordhere!";

### Delete previous versions of the report

$output_name="> ${output_dir}/${report_name}";

open(OUTPUT,$output_name);


### Get list of indexes
@index_config_raw=`cat /opt/splunk/etc/system/local/indexes.conf`;

foreach $line (@index_config_raw) {
        chomp $line;

        if ($line=~m/\[/) {
                $line=~m/\[(.*)\]/g;
                $raw_index=$1;

                push(@indexes,$raw_index);

        }
}

print OUTPUT "$header \n";

### Processes indexes
foreach $index (@indexes) {
        chomp $index;

        $splunk_command="${splunk_bin_dir}/splunk search \"| dbinspect index=\"${index}\" timeformat=\"\%s\" | rename state as category | stats min(earliestTime) as earliestTime max(latestTime) as latestTime sum(sizeOnDiskMB) as MB by category | convert timeformat=\"\%m/\%d/\%Y\" ctime(earliestTime) as earliestTime ctime(latestTime) as latestTime\" -auth ${username}:${password}| grep -v \"category\" | grep -v \"-\" ";


        @result=`${splunk_command}`;

        if ($#result ne "-1") {
                foreach $return (@result) {
                        chomp $return;

                        $return=~m/(hot|warm|cold|frozen)\s+([\d]+\/[\d]+\/[\d]+)\s+([\d]+\/[\d]+\/[\d]+)\s+([\d]+\.[\d]+)/gi;

                        $db_type=$1;
                        $earliest_event=$2;
                        $latest_event=$3;
                        $size=$4;

                        print OUTPUT "$index,$db_type,$earliest_event,$latest_event,$size \n";


                }
        }

        if ($#result eq "-1") {
                print OUTPUT "$index,Empty Index,Empty Index,Empty Index \n";
        }


}

if ($mail_to ne "") {
        `uuencode ${output_dir}/${report_name} ${output_dir}/${report_name} | mailx -s \"Splunk Index Report\" $mail_to`;
}
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...