Splunk Search

splunk search for index's maximum configured size

w531t4
Path Finder

Hi all,

I found an answer here on the Splunk forums that shows a good search to list the current size of indexes as they sit on disk.. I would now like to associate these numbers with the MB size restrictions i have configured in indexes.conf per index..

Does anyone know of a good search that would produce theses values?

0 Karma
1 Solution

w531t4
Path Finder

"|btool indexes" definitely was the way to go. This is really what i was looking for.

| btool indexes 
| rex mode=sed "s/\r?\n/--BREAKER--/g" 
| rex field=_raw "(?<firstline>.+?)--BREAKER--(?<otherlines>.*)$" 
| eval otherlines=split(otherlines, "--BREAKER--")
| rex field=firstline ".*?\s+\[(?<indexname>.+)\]$"
| rex field=otherlines "(?<a>\S+)\s+(?<b>[^=]+)=(?<c>.*)" max_match=1 
| eval fields=mvzip(a,mvzip(b,c))
| mvexpand fields 
| rex field=fields "^(?<filename>[^,]+),(?<k>[^,]+),(?<v>.*)" 
| table filename,k,v,sos_server,indexname
| where k like "%maxTotalDataSizeMB%"

View solution in original post

w531t4
Path Finder

"|btool indexes" definitely was the way to go. This is really what i was looking for.

| btool indexes 
| rex mode=sed "s/\r?\n/--BREAKER--/g" 
| rex field=_raw "(?<firstline>.+?)--BREAKER--(?<otherlines>.*)$" 
| eval otherlines=split(otherlines, "--BREAKER--")
| rex field=firstline ".*?\s+\[(?<indexname>.+)\]$"
| rex field=otherlines "(?<a>\S+)\s+(?<b>[^=]+)=(?<c>.*)" max_match=1 
| eval fields=mvzip(a,mvzip(b,c))
| mvexpand fields 
| rex field=fields "^(?<filename>[^,]+),(?<k>[^,]+),(?<v>.*)" 
| table filename,k,v,sos_server,indexname
| where k like "%maxTotalDataSizeMB%"

aholzer
Motivator

Very cool. Didn't realize the btool function was available at search time. Thought it was just a CLI thing. Glad you found the answer 🙂

0 Karma

w531t4
Path Finder

gotcha -- looks like |btool indexes is the way to go!

0 Karma

aholzer
Motivator

Well... you could write a script that monitors your indexes.conf, and aggregates the max_sizes for you and then gets indexed in say main. Then you could simply search against that data rather than maintaining a lookup.

Or you could write a script that writes your lookup csv and runs automatically on a schedule, therefore removing the need for manual intervention.

There may be a way of doing it out of the box, it just escapes me. There are people far more knowledgable than me on here though, and one of them might take a look at your question and chime in with a brilliant answer 🙂

0 Karma

w531t4
Path Finder

OK - I suppose i was originally hoping that I could pull the configuration values out rather than maintaining an the index sizes in both indexes.conf and a lookup csv. But, i guess it is what it is. Thanks for the help.

0 Karma

aholzer
Motivator

I'd suggest you create a csv with two columns: index and max_size

Then use lookup GUI interface to create a lookup table and definition with this data (index_size.csv and index_size). You can then use the lookup command to get the max_size from the table and link it with your search. Like so:

| eventcount summarize=false report_size=true index=*
| eval MB = size_bytes / 1024 / 1024
| lookup index_size index OUTPUT max_size

This will then append the column "max_size" from your file to your results.

Hope this helps

0 Karma

w531t4
Path Finder

my fault.

| eventcount summarize=false report_size=true index=*
| eval MB = size_bytes / 1024 / 1024

0 Karma

aholzer
Motivator

Going forward it would be helpful if you add a link to the other answer you found, or put the search you want to upgrade in your question.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...