topic Can you help me craft a search that returns all indexes with their associated retention times? in Getting Data In

Can you help me craft a search that returns all indexes with their associated retention times?

awmorris — Thu, 06 Sep 2018 13:49:16 GMT

Technically, this is two questions in one with the goal of solving a single problem: I need an SPL query that returns ALL the indexes I can search and the associated retention time for each. Here is how far I've gotten:

| rest /services/data/indexes | eval yr = floor(frozenTimePeriodInSecs/86400/365)| eval dy = (frozenTimePeriodInSecs/86400) % 365 | eval ret = yr . " years, " . dy . " days" | stats list(splunk_server) list(frozenTimePeriodInSecs) list(ret) by title

The query above is very very close, but it only returns a subset of the indexes — technically, it only returns 32 index names to me, and I have many more than that. (Note- starting with "rest /services/admin/indexes ... " makes no difference either.

My second query is this:

| eventcount summarize=false index=* index=_* | dedup index | fields index

That will return all 250+ index names, but I can't seem to find anyway to get back to the retention period.

So my two questions are:
1) Why is the rest command only pulling a subset (<15%) of all indexes that are returned by the event count query?
2) How can I get a single query that gets to my goal to have a single SPL query that shows all 250+ indexes and their associated retention setting?

Re: Can you help me craft a search that returns all indexes with their associated retention times?

inventsekar — Thu, 06 Sep 2018 14:50:36 GMT

| rest /services/data/indexes -
technically, it only returns 32 index names to me and i have many more than that ?!?!?
may i know why it returns only 32?!?! on my splunk, it returns more than 2000 indexes.

Re: Can you help me craft a search that returns all indexes with their associated retention times?

adonio — Thu, 06 Sep 2018 14:53:34 GMT

tried your top search and it works nice
can you double check?
or try this and use your evals
| rest /services/data/indexes-extended
| table title frozenTimePeriodInSecs

Re: Can you help me craft a search that returns all indexes with their associated retention times?

renjith_nair — Thu, 06 Sep 2018 14:58:14 GMT

@awmorris,

By default, maximum number of entries returned is 30. Please refer to this documentation :

Maximum number of entries to return. Set value to 0 to get all available entries.

http://docs.splunk.com/Documentation/Splunk/7.1.2/RESTREF/RESTprolog#Pagination_and_filtering_parameters

Try

| rest /services/data/indexes count=0 to override the default vlaue

Re: Can you help me craft a search that returns all indexes with their associated retention times?

awmorris — Thu, 06 Sep 2018 17:01:01 GMT

this is the exact scenario i am facing.

Re: Can you help me craft a search that returns all indexes with their associated retention times?

awmorris — Thu, 06 Sep 2018 17:02:54 GMT

Good call out on the count limit.... but i still only get 32. 😞

Re: Can you help me craft a search that returns all indexes with their associated retention times?

awmorris — Thu, 06 Sep 2018 17:03:37 GMT

I reran it again.... even this simple query ONLY returns 32 indexes:

"| rest /services/data/indexes-extended | table title frozenTimePeriodInSecs"

Re: Can you help me craft a search that returns all indexes with their associated retention times?

awmorris — Thu, 06 Sep 2018 18:41:53 GMT

Updated info: If I use an account with Administrator privileges, I get the full list- not just the 32- so it must be a permission thing somehow.

Re: Can you help me craft a search that returns all indexes with their associated retention times?

renjith_nair — Fri, 07 Sep 2018 02:34:32 GMT

@awmorris,

Also check the permissions as mentioned in the doc http://docs.splunk.com/Documentation/Splunk/7.1.2/RESTREF/RESTintrospect#data.2Findexes

**Authorization and authentication**
By default, all users can list all indexes. However, if the indexes_list_all capability is enabled in authorize.conf, access to all indexes is limited to only those roles with this capability.

To enable indexes_list_all capability restrictions on the data/indexes endpoint, create a [capability::indexes_list_all] stanza in authorize.conf. Specify indexes_list_all=enabled for any role permitted to list all indexes from this endpoint.

Re: Can you help me craft a search that returns all indexes with their associated retention times?

bandit — Mon, 25 Nov 2019 03:41:47 GMT

Dashboard to interactively review index parameters and their values:

<form theme="dark">
  <label>Indexes</label>
  <fieldset submitButton="false">
    <input type="text" token="title_pattern" searchWhenChanged="true">
      <label>Index Pattern</label>
      <default>*</default>
    </input>
    <input type="text" token="title_list" searchWhenChanged="true">
      <label>Index List (,separated)</label>
      <default>*</default>
    </input>
    <input type="dropdown" token="title" searchWhenChanged="true">
      <label>Index</label>
      <choice value="*">All</choice>
      <fieldForLabel>title</fieldForLabel>
      <fieldForValue>title</fieldForValue>
      <search>
        <query>| rest /services/data/indexes 
| search title="*$title_pattern$*" title IN($title_list$)
| dedup title
| table title 
| sort title</query>
        <earliest>-1m</earliest>
        <latest>now</latest>
      </search>
      <default>*</default>
    </input>
    <input type="text" token="parameter_pattern" searchWhenChanged="true">
      <label>Parameter Pattern</label>
      <default>*</default>
    </input>
    <input type="text" token="parameter_list" searchWhenChanged="true">
      <label>Parameter List  (,separated)</label>
      <default>*</default>
    </input>
    <input type="dropdown" token="parameter" searchWhenChanged="true">
      <label>Parameter</label>
      <choice value="*">All</choice>
      <default>*</default>
      <fieldForLabel>parameter</fieldForLabel>
      <fieldForValue>parameter</fieldForValue>
      <search>
        <query>| rest /services/data/indexes 
| search title=$title$ title="*$title_pattern$*" title IN($title_list$) 
| transpose 0 column_name="parameter" header_field="title" 
| search parameter="*$parameter_pattern$*" parameter IN($parameter_list$) NOT parameter="parameter"
| table parameter</query>
        <earliest>-1m</earliest>
        <latest>now</latest>
      </search>
    </input>
    <input type="text" token="value_pattern" searchWhenChanged="true">
      <label>Value Pattern</label>
      <default>*</default>
    </input>
    <input type="text" token="value" searchWhenChanged="true">
      <label>Value (exact)</label>
      <default>*</default>
    </input>
  </fieldset>
  <row>
    <panel>
      <title>Index Parameters</title>
      <table>
        <search>
          <query>| rest /services/data/indexes 
| search title=$title$ title="*$title_pattern$*" title IN($title_list$) 
| transpose 0 column_name="parameter" header_field="title" 
| search parameter="*$parameter_pattern$*" parameter IN($parameter_list$) parameter="$parameter$" 
| untable parameter title value 
| table title parameter value 
| eval {parameter}=value 
| search value="*$value_pattern$*" value="$value$" 
| table title parameter value 
| chart limit=1000000 values(value) as value by parameter title</query>
          <earliest>-30m@m</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">15</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">none</option>
        <option name="percentagesRow">false</option>
        <option name="refresh.display">progressbar</option>
        <option name="rowNumbers">true</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
      </table>
    </panel>
  </row>
</form>

Re: Can you help me craft a search that returns all indexes with their associated retention times?

woodcock — Mon, 25 Nov 2019 14:57:34 GMT

There is a search on the Monitoring Console that gives you most of this but it is missing a piece: how to see what your actual effective retention is based on the buckets that are freezing. You can see that part with this search:

index="_internal" AND sourcetype="splunkd" AND bucketmover AND freeze 
| rex "[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db[\/\\\]db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+" 
| rex "db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+.*?[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db" 
| sort 0 indexname - oldestTime
| dedup indexname 
| eval retention = _time - oldestTime 
| fieldformat retention = tostring(retention, "duration") 
| table _time indexname retention