Issue trying to archive Splunk data

[uayub]

The following indexes.conf was created in the local directory (in Ubuntu).

index specific defaults

maxDataSize = 90000
maxHotSpanSecs = 7776000
maxWarmDBCount = 2
frozenTimePeriodInSecs = 86400
coldToFrozenDir = /opt/splunk/var/lib/splunk/defaultdb/frozendb

The expectection was that the Hot bucket would keep 90 GB of data (which would be around 90 days) before moving it to Warm (either via time or size). There would be just 2 Warm buckets (around 6 months of data). As soon as another Hot buckeet would roll to Warm, the old Warm bucket would move to Cold. And stay over here for at least 24 hours.

But as soon as the above change was implemented adn Splunk daemon restarted, I could perform a Search just for a day. All the data was moved to the Frozen folder.

Questions:
1) why did this happen? Supposedly 6 months of data should have been in Hot+Warm buckets.
2) Main goal was to keep 180 days of data in Hot+Warm and I should be able to search 1 year of data (for PCI).
3) If I were to restore my data from the Frozen folder, is there a way to restore multiple files at at time? I found an article which shows how to restore 1 file at a time.

Thanks

UA

[bmacias84]

Hello @uayub,

Keep in mind that those are not the only settings. is your maxDataSize Greater than maxTotalDataSizeMB. Not quite sure why you are secifying maxWarmDBCount. I use the default maxWarmDBCount for preformance reasons.

1. maxTotalDataSizeMB (default is 500GB)
2. Did you set maxHotIdleSecs (should be zero)
3. But the Culprit seems to be your frozenTimePeriodInSecs = 86400

frozenTimePeriodInSecs will move every event in the DB (INDEX) older than value set and will be frozen when splunkd checks in based on rotatePeroidInSecs attribute. ReRead the indexes.conf on Splunk Base

Also use the btool for debugging your conf file. $SPLUNK_HOME/bin/splunk btool --debug indexese list. This will dump all your index .conf with which app is settings them.

Hope this help or gets you started. If this does dont forget to accept or vote up.

Cheers,

[bmacias84]

@uayub,
I see you voted up my answer and I hope it did. Would you mind accepting the anwser by clicking the checkmark? Thanks.

cheers