Hi,
the size of my Splunk database is at around >1TB+.
I would like to know about all available Indexes and especially all of the associated SourceTypes
and the amount of it.
The search in WebUI works no problem for the last 24hrs but searching for all of the data
takes forever and times out.
I'm aware that saved searches would be an option but i'm curious to know if a script
would work which recursive scans the database and process all SourceTypes.data file
like
< /opt/splunk/var/lib/splunk/sampledb/db/db_1680195600_1672423200_0/SourceTypes.data
< /opt/splunk/var/lib/splunk/sampledb/db/db_1698782400_1680199200_1/SourceTypes.data
...
...
Would this be a feasable option?
Many thanks
Hi @MattKr,
Here's an option that will run from the UI.
| rest /services/data/indexes splunk_server=local
| stats count by title | rename title as index
| map [| metadata type=sourcetypes index=$index$ | eval index="$index$"] maxsearches=100
This uses the metadata command to get the sourcetypes, and earliest/latest times, and the number of matching events. The one drawback is that the index isn't included in the results, so I've set it up via the map command so it will run the metadata search for each index.
Couple of things to note:
I ran the search on a small cloud environment with 52 indexes over all time and it completed in 4.9s.
Give that a go.
Hi @MattKr,
Here's an option that will run from the UI.
| rest /services/data/indexes splunk_server=local
| stats count by title | rename title as index
| map [| metadata type=sourcetypes index=$index$ | eval index="$index$"] maxsearches=100
This uses the metadata command to get the sourcetypes, and earliest/latest times, and the number of matching events. The one drawback is that the index isn't included in the results, so I've set it up via the map command so it will run the metadata search for each index.
Couple of things to note:
I ran the search on a small cloud environment with 52 indexes over all time and it completed in 4.9s.
Give that a go.
Hi danspav,
thank you so much, the query took around 300 sec. on around 10 indexes, 4TB db size and returns what i'm looking for, perfect!
@MattKr What is your retention period of logs? Also you can look at _internal logs by sourcetype to get the required data but these internal logs are stored only for 30days by default.