Why am I unable to retrieve data prior to the date our archive directory disk was filled?

[trumpjk]

I am running searches on a Splunk server and am unable to retrieve data prior to the date our disk filled. I corrected the space issues but when I execute a search, it stops at the point where the filesystem filled and does not retrieve data prior to that date when there was not a space issue. The space issue was caused by the archive directory residing on the same disk as the live indexes. This was corrected. Any suggestions?

[lycollicott · Accepted solution]

UPDATE: The index reached its maximum size and that is why the buckets were frozen. Increase the size of the index by increasing the value of maxTotalDataSizeMB to prevent it in the future.

ORIGINAL:
A bucket gets put into coldToFrozenDir once frozenTimePeriodInSecs is reached and it is immediately unsearchable until you thaw it. (http://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/Restorearchiveddata )

Can you post one of your bucket names in coldToFrozenDir?

Here is one of mine:

db_1476989749_1476981284_3458_CACEB811-4B3C-4B60-AE46-A061185F4F10

Bucket names are defined into 5 "fields" separated by "_" like this:
db = a local bucket created on the local machine
1476989749 = epoch time representing the most recent event in the bucket
1476981284 = epoch time representing the oldest event in the bucket
3458 = bucket sequence id
CACEB811-4B3C-4B60-AE46-A061185F4F10 = the GUID of your Splunk instance

For this bucket, it contains events from:
20 Oct 2016 16:34:44
to:
20 Oct 2016 18:55:49

If your bucket name translates to a date/time before 12/12 then you will not be able to search it until you thaw it.

UPDATE: I don't think there is anything special about 12/12 for you....that just happens to be the day the buckets froze and you moving them to another location is purely coincidence.

You can use http://www.epochconverter.com/ to figure out what the epoch times translate to.

[trumpjk]

It is just interesting that data was frozen that should not have been frozen. I am in the process of restoring frozen buckets to get the data back. Doesn't seem to be an easy way for me to restore data between certain dates, more of a manual process.

[lycollicott]

Can you post a bucket name?

[trumpjk]

db_1481231014_1473455015_2404

[lycollicott]

Do you have more than one setting for your index?

Run:
splunk btool indexes list <your_index_name> --debug

[trumpjk]

rozenTimePeriodInSecs only appears in one place and is set to 18396000

[lycollicott]

Can you post output from this for 12/5 to 12/12 ?

index="_internal" component=BucketMover db_1481231014_1473455015_2404

[trumpjk]

10:53:28.564 AM
12-19-2016 10:53:28.564 -0500 INFO BucketMover - AsyncFreezer freeze succeeded for bkt='/export/splunk/var/lib/splunk/defaultdb/db/db_1481231014_1473455015_2404'
eventtype = nix-all-logs eventtype = splunkd-log host = splog.jhuapl.edu index = _internal source = /export/splunk/var/log/splunk/splunkd.log sourcetype = splunkd

10:53:28.198 AM
12-19-2016 10:53:28.198 -0500 INFO BucketMover - will attempt to freeze bkt='/export/splunk/var/lib/splunk/defaultdb/db/db_1481231014_1473455015_2404' because maxTotalDataSize=524288000000 bytes, currentSize=524476821504 bytes

[lycollicott]

There is your explanation right there. It was frozen as a result if size not its age.

[trumpjk]

What setting controls the size?

I currently have: maxTotalDataSizeMB = 700000

[lycollicott]

From http://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/Indexesconf :

maxTotalDataSizeMB =
* The maximum size of an index (in MB).
* If an index grows larger than the maximum size, the oldest data is frozen.
* This parameter only applies to hot, warm, and cold buckets. It does not
apply to thawed buckets.
* Highest legal value is 4294967295
* Defaults to 500000.

[trumpjk]

Thanks for your help.

[lycollicott]

No problem and you're welcome. I work for Karma points LOL

[lycollicott]

Exactly how did you correct it? Did you stop Splunk, relocate the files, edit indexes.conf then restart Splunk?

[trumpjk]

I stopped splunk. Renamed the archive directory, mounted an NFS share to the specified archive_directory in indexes.conf, copied files from the renamed archive directory to the correct location, restarted splunk.

[lycollicott]

Ok, when you say "archive" do you mean the "Frozen Path"? Something like this in indexes.conf?

[wineventlog]
coldToFrozenDir = R:\splunkdb\wineventlog\frozendb

If that is the case then frozen data cannot be read unless thawed first.

[trumpjk]

Yes, coldToFrozenDir. Data in question should not be frozen as frozenTimePeriodInSecs = 18396000 which is approx 212 days. the disk issue happened on 12/12. When I search I get no data older than 12/12.