Splunk arbitrarily deletes index on restart

[redc]

I have one particular index whose data gets deleted any time Splunk is restarted. I see this in the splunkd.log:

idx=my_index Removing; IP::deleteIndex
idx=my_index Removing; wait for in-flights
idx=my_index Removing; erasing DPP from lookups
idx=my_index Handling shutdown or signal, reason=3
idx=my_index Deletion approved, start dir removal
closing hot mgr for idx=my_index
idx=my_index Removing; erased directory='E:\Splunk\var\lib\splunk\my_index\db' (param=homePath)
idx=my_index Removing; erased directory='E:\Splunk\var\lib\splunk\my_index\colddb' (param=coldPath)
idx=my_index Removing; parameter=bloomHomePath has no assigned value
idx=my_index Removing; directory='E:\Splunk\var\lib\splunk\my_index\summary' (param=summaryHomePath) not found
idx=my_index Removing; parameter=tstatsHomePath has no assigned value
idx=my_index Removing; erased directory='E:\Splunk\var\lib\splunk\my_index\thaweddb' (param=thawedPath)
idx=my_index Removing; erased directory='E:\Splunk\var\lib\splunk\my_index' (param=index proper)
removing index=my_index stanza from indexes.conf app=my_app
idx=my_index Finished removing

When Splunk starts back up again, it goes through the process of creating everything that was just deleted.

As far as I can tell, there are no differences between this index and any other index. It was created through the Admin with the default settings, exactly the same way as every other index I've created. There's nothing special about the data; we're indexing similar data in two other indexes which are unaffected by Splunk restarts (.csv data).

Is there some setting hidden in a .conf file somewhere that I should look for?

[redc]

@mkinsley_splunk:

We deleted, restarted, and recreated the index. We also updated the demo data to use more recent data.

That seems to have fixed the issue, except...now one of the other indexes (also starting with "quick_rad_" and with older data) is now exhibiting this behavior.

[lukejadamec · Accepted solution]

The default for the age of data before it is frozen (deleted) is frozenTimePeriodInSecs = 188697600, which is about 6 years. In order for a bucket to get deleted, all data in the bucket must be older than the setting (about 6 years). If the other indexes buckets also contain newer data (less than 6 years) then they would not be deleted.

Try including some recent inputs into the index.

You can check the newest and oldest event in a bucket by looking at the bucket name. The two long numbers are epoch time stamps for the newest and oldest event in the bucket.

[redc]

We set a frozen archive path on the affected index, re-imported the .csv data, and then restarted Splunk and it did create the frozen database archive under the frozen archive path.

We'll just have to manually manipulate the demo data our program is spitting out to have more recent dates.

[redc]

We'll try that, but the other "quick_rad_" indexes don't have newer data in them, either, and they're not being affected.

[redc]

@lguinn:

1. "quick_rad_orderhistory". The unaffected indexes all start with "quick_rad_".

2. No, no special characters or underscores or anything at the start of the index names.

[redc]

@lukejadamec:

1. The data falls between 2001 and 2004 (it's demo data generated by a really old program). There is other data in the same date range that is in other indexes and is unaffected, so I don't think it's age-related.

2. Less than 1MB (305 events). Related data is between 140 and 1,000 events (all under 1MB).

3. No, we haven't changed the properties manually.

4. Just the file structure. That's why it's so frustrating because we then have to re-import the data.

[redc]

@mkinsley_splunk:

1. The log lines above came from _internal. It appears to be initiated by the Splunk service restart.

2. I'll give that a shot as soon as I can schedule it with the person using the data.

[lguinn2]

What did you name the index? Does the name begin with an underscore?

[lukejadamec]

What are the timestamps for the data you're sending to this index?
What is the size of the data?
Have you changed the default properties in etc/system/local/indexes.conf or default/indexes.conf?
When you say 'recreate what was deleted' do you mean the file structure or the actual data?

[mkinsley_splunk]

Two questions:

1. Can you do a more general search in _index to see if there is anything specific initiating the request to delete the index?

2. If you find no root cause in 1) , what happens if you actively delete the index, then restart, then recreate the index with the same name. Does the behavior persist?