topic Re: Splunk arbitrarily deletes index on restart in Deployment Architecture

Splunk arbitrarily deletes index on restart

redc — Wed, 29 Jan 2014 22:35:29 GMT

I have one particular index whose data gets deleted any time Splunk is restarted. I see this in the splunkd.log:

idx=my_index Removing; IP::deleteIndex
idx=my_index Removing; wait for in-flights
idx=my_index Removing; erasing DPP from lookups
idx=my_index Handling shutdown or signal, reason=3
idx=my_index Deletion approved, start dir removal
closing hot mgr for idx=my_index
idx=my_index Removing; erased directory='E:\Splunk\var\lib\splunk\my_index\db' (param=homePath)
idx=my_index Removing; erased directory='E:\Splunk\var\lib\splunk\my_index\colddb' (param=coldPath)
idx=my_index Removing; parameter=bloomHomePath has no assigned value
idx=my_index Removing; directory='E:\Splunk\var\lib\splunk\my_index\summary' (param=summaryHomePath) not found
idx=my_index Removing; parameter=tstatsHomePath has no assigned value
idx=my_index Removing; erased directory='E:\Splunk\var\lib\splunk\my_index\thaweddb' (param=thawedPath)
idx=my_index Removing; erased directory='E:\Splunk\var\lib\splunk\my_index' (param=index proper)
removing index=my_index stanza from indexes.conf app=my_app
idx=my_index Finished removing

When Splunk starts back up again, it goes through the process of creating everything that was just deleted.

As far as I can tell, there are no differences between this index and any other index. It was created through the Admin with the default settings, exactly the same way as every other index I've created. There's nothing special about the data; we're indexing similar data in two other indexes which are unaffected by Splunk restarts (.csv data).

Is there some setting hidden in a .conf file somewhere that I should look for?

Re: Splunk arbitrarily deletes index on restart

mkinsley_splunk — Wed, 29 Jan 2014 23:04:48 GMT

Two questions:

Can you do a more general search in _index to see if there is anything specific initiating the request to delete the index?
If you find no root cause in 1) , what happens if you actively delete the index, then restart, then recreate the index with the same name. Does the behavior persist?

Re: Splunk arbitrarily deletes index on restart

lukejadamec — Thu, 30 Jan 2014 01:00:25 GMT

What are the timestamps for the data you're sending to this index?
What is the size of the data?
Have you changed the default properties in etc/system/local/indexes.conf or default/indexes.conf?
When you say 'recreate what was deleted' do you mean the file structure or the actual data?

Re: Splunk arbitrarily deletes index on restart

lguinn2 — Thu, 30 Jan 2014 06:15:55 GMT

What did you name the index? Does the name begin with an underscore?

Re: Splunk arbitrarily deletes index on restart

redc — Thu, 30 Jan 2014 16:01:59 GMT

@mkinsley_splunk:

The log lines above came from _internal. It appears to be initiated by the Splunk service restart.
I'll give that a shot as soon as I can schedule it with the person using the data.

Re: Splunk arbitrarily deletes index on restart

redc — Thu, 30 Jan 2014 16:03:05 GMT

@lukejadamec:

The data falls between 2001 and 2004 (it's demo data generated by a really old program). There is other data in the same date range that is in other indexes and is unaffected, so I don't think it's age-related.
Less than 1MB (305 events). Related data is between 140 and 1,000 events (all under 1MB).
No, we haven't changed the properties manually.
Just the file structure. That's why it's so frustrating because we then have to re-import the data.

Re: Splunk arbitrarily deletes index on restart

redc — Mon, 28 Sep 2020 15:46:27 GMT

@Anonymous:

"quick_rad_orderhistory". The unaffected indexes all start with "quick_rad_".
No, no special characters or underscores or anything at the start of the index names.

Re: Splunk arbitrarily deletes index on restart

lukejadamec — Thu, 30 Jan 2014 16:13:23 GMT

The default for the age of data before it is frozen (deleted) is frozenTimePeriodInSecs = 188697600, which is about 6 years. In order for a bucket to get deleted, all data in the bucket must be older than the setting (about 6 years). If the other indexes buckets also contain newer data (less than 6 years) then they would not be deleted.

Try including some recent inputs into the index.

You can check the newest and oldest event in a bucket by looking at the bucket name. The two long numbers are epoch time stamps for the newest and oldest event in the bucket.

Re: Splunk arbitrarily deletes index on restart

redc — Mon, 28 Sep 2020 15:46:32 GMT

We'll try that, but the other "quick_rad_" indexes don't have newer data in them, either, and they're not being affected.

Re: Splunk arbitrarily deletes index on restart

redc — Mon, 28 Sep 2020 15:46:40 GMT

@mkinsley_splunk:

We deleted, restarted, and recreated the index. We also updated the demo data to use more recent data.

That seems to have fixed the issue, except...now one of the other indexes (also starting with "quick_rad_" and with older data) is now exhibiting this behavior.

Re: Splunk arbitrarily deletes index on restart

redc — Thu, 30 Jan 2014 21:00:50 GMT

We set a frozen archive path on the affected index, re-imported the .csv data, and then restarted Splunk and it did create the frozen database archive under the frozen archive path.

We'll just have to manually manipulate the demo data our program is spitting out to have more recent dates.