Deployment Architecture

Splunk arbitrarily deletes index on restart

Builder

I have one particular index whose data gets deleted any time Splunk is restarted. I see this in the splunkd.log:

idx=my_index Removing; IP::deleteIndex
idx=my_index Removing; wait for in-flights
idx=my_index Removing; erasing DPP from lookups
idx=my_index Handling shutdown or signal, reason=3
idx=my_index Deletion approved, start dir removal
closing hot mgr for idx=my_index
idx=my_index Removing; erased directory='E:\Splunk\var\lib\splunk\my_index\db' (param=homePath)
idx=my_index Removing; erased directory='E:\Splunk\var\lib\splunk\my_index\colddb' (param=coldPath)
idx=my_index Removing; parameter=bloomHomePath has no assigned value
idx=my_index Removing; directory='E:\Splunk\var\lib\splunk\my_index\summary' (param=summaryHomePath) not found
idx=my_index Removing; parameter=tstatsHomePath has no assigned value
idx=my_index Removing; erased directory='E:\Splunk\var\lib\splunk\my_index\thaweddb' (param=thawedPath)
idx=my_index Removing; erased directory='E:\Splunk\var\lib\splunk\my_index' (param=index proper)
removing index=my_index stanza from indexes.conf app=my_app
idx=my_index Finished removing

When Splunk starts back up again, it goes through the process of creating everything that was just deleted.

As far as I can tell, there are no differences between this index and any other index. It was created through the Admin with the default settings, exactly the same way as every other index I've created. There's nothing special about the data; we're indexing similar data in two other indexes which are unaffected by Splunk restarts (.csv data).

Is there some setting hidden in a .conf file somewhere that I should look for?

Tags (2)
0 Karma
1 Solution

Super Champion

The default for the age of data before it is frozen (deleted) is frozenTimePeriodInSecs = 188697600, which is about 6 years. In order for a bucket to get deleted, all data in the bucket must be older than the setting (about 6 years). If the other indexes buckets also contain newer data (less than 6 years) then they would not be deleted.

Try including some recent inputs into the index.

You can check the newest and oldest event in a bucket by looking at the bucket name. The two long numbers are epoch time stamps for the newest and oldest event in the bucket.

View solution in original post

0 Karma

Builder

@mkinsley_splunk:

We deleted, restarted, and recreated the index. We also updated the demo data to use more recent data.

That seems to have fixed the issue, except...now one of the other indexes (also starting with "quick_rad_" and with older data) is now exhibiting this behavior.

0 Karma

Super Champion

The default for the age of data before it is frozen (deleted) is frozenTimePeriodInSecs = 188697600, which is about 6 years. In order for a bucket to get deleted, all data in the bucket must be older than the setting (about 6 years). If the other indexes buckets also contain newer data (less than 6 years) then they would not be deleted.

Try including some recent inputs into the index.

You can check the newest and oldest event in a bucket by looking at the bucket name. The two long numbers are epoch time stamps for the newest and oldest event in the bucket.

View solution in original post

0 Karma

Builder

We set a frozen archive path on the affected index, re-imported the .csv data, and then restarted Splunk and it did create the frozen database archive under the frozen archive path.

We'll just have to manually manipulate the demo data our program is spitting out to have more recent dates.

0 Karma

Builder

We'll try that, but the other "quick_rad_" indexes don't have newer data in them, either, and they're not being affected.

0 Karma

Builder

@lguinn:

  1. "quick_rad_orderhistory". The unaffected indexes all start with "quick_rad_".

  2. No, no special characters or underscores or anything at the start of the index names.

0 Karma

Builder

@lukejadamec:

  1. The data falls between 2001 and 2004 (it's demo data generated by a really old program). There is other data in the same date range that is in other indexes and is unaffected, so I don't think it's age-related.

  2. Less than 1MB (305 events). Related data is between 140 and 1,000 events (all under 1MB).

  3. No, we haven't changed the properties manually.

  4. Just the file structure. That's why it's so frustrating because we then have to re-import the data.

0 Karma

Builder

@mkinsley_splunk:

  1. The log lines above came from _internal. It appears to be initiated by the Splunk service restart.

  2. I'll give that a shot as soon as I can schedule it with the person using the data.

0 Karma

Legend

What did you name the index? Does the name begin with an underscore?

0 Karma

Super Champion

What are the timestamps for the data you're sending to this index?
What is the size of the data?
Have you changed the default properties in etc/system/local/indexes.conf or default/indexes.conf?
When you say 'recreate what was deleted' do you mean the file structure or the actual data?

0 Karma

Splunk Employee
Splunk Employee

Two questions:

  1. Can you do a more general search in _index to see if there is anything specific initiating the request to delete the index?

  2. If you find no root cause in 1) , what happens if you actively delete the index, then restart, then recreate the index with the same name. Does the behavior persist?

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!