Retention Time Question

[hartfoml]

I have this setting in my index.conf

frozenTimePeriodInSecs = 48211200

As I understand it this should keep data in the index for about 1.53 years

I do not have any data in the index past January 9th of this year. Somehow the data has rolled out or has it?

Also if the data really has rolled out, then I need to restore from an old copy of the index back from January.

Can you point me to the process doc to connect an old copy of the index and extract the logs (Many Millions of lines - Firewall Logs) and put the logs back into the online index.

Thanks for the help.

Support said they wouldn't be able to help until tomorrow and needed a diag.

A diag? for what, to find out what happened 11 months ago?

How would I check to see what is rolling out of the firewall index?

[alacercogitatus · Accepted solution]

The file should be "indexes.conf". Where is this file located?

You can check the settings by using: $SPLUNK_HOME/bin/splunk cmd btool --debug indexes list YOUR_INDEX_NAME. This will show you from what file a setting was included, and what all the options are for that index. Do this on the indexers.

If you have really rolled data - you will need to restore the buckets - copy and paste will do as long as the bucket ids don't collide.

EDIT AND UPDATE:

So according to your last comment - you are maxing your index at <1TB, so it will roll off data when the index reaches that threshold. Set your maxTotalDataSizeMB = 104857600. You will then start storing your data up to 1.5 years OR 100 TB, which ever comes FIRST.

[hartfoml]

Kyle,

You mentoend above that after resoring the buckets I could copy to the db and as long as the bucket ID's don't collide.

Is that right? or do I need to do a ./splunk rebuild $SPLUNK_HOME/var/lib/splunk/firewall/thaweddb/temp_db_1*

And then copy reindexed temp folder to $SPLUNK_HOME/var/lib/splunk/firewall/db/db_1*

[alacercogitatus]

See my original answer for the answer. You are limiting based on size, not time. Update the conf for the index and it should start storing all the data.

[hartfoml]

There are no files in the $SPLUNK_DB/firewall/colddb

[splunk@Mysystem splunk]$ du -sh firewall/

978G firewall/

I have three indexers with the same size

[alacercogitatus]

also just hit me, you have "maxTotalDataSizeMB = 1000000", which is less than 1 TB. Not sure about you, but a years worth of firewall data is more than that. What does du -sh $SPLUNK_DB/firewall return?

[alacercogitatus]

COLDDB - not DB

[hartfoml]

in the $SPLUNK_DB/firewall/db I have these files

drwx------ 3 splunk splunk 8192 Jan 15 2013 db_1357863050_1357725658_118
drwx------ 3 splunk splunk 4096 Jan 17 2013 db_1358072144_1357863051_119
drwx------ 3 splunk splunk 4096 Jan 18 2013 db_1358267635_1358072147_120
drwx------ 3 splunk splunk 8192 Jan 21 2013 db_1358415963_1358267682_121
drwx------ 3 splunk splunk 4096 Jan 23 2013 db_1358556654_1358415964_122
drwx------ 3 splunk splunk 4096 Jan 24 2013 db_1358773965_1358556655_123
drwx------ 3 splunk splunk 8192 Jan 26 2013 db_1358945916_1358773976_12

[alacercogitatus]

Try these commands and see what returns (run them from $SPLUNK_DB/firewall/colddb😞

touch --date "2013-01-02" /tmp/start
touch --date "2013-01-03" /tmp/end
find . -type f -newer /tmp/start -not -newer /tmp/end

[hartfoml]

Search GUI

[alacercogitatus]

This shows that your frozen time is correct. How are you determining that the data is not there? Via Search GUI or manually looking in the db files?

[hartfoml]

system rawChunkSizeBytes = 131072
system rotatePeriodInSecs = 60
system serviceMetaPeriod = 25
system suppressBannerList =
system sync = 0
system syncMeta = true
system thawedPath = $SPLUNK_DB/firewall/thaweddb
system throttleCheckPeriod = 15

[hartfoml]

system maxConcurrentOptimizes = 3
system maxDataSize = auto_high_volume
system maxHotBuckets = 3
system maxHotIdleSecs = 0
system maxHotSpanSecs = 7776000
system maxMemMB = 5
system maxMetaEntries = 1000000
system maxRunningProcessGroups = 20
system maxRunningProcessGroupsLowPriority = 1
system maxTotalDataSizeMB = 1000000
system maxWarmDBCount = 300
system memPoolMB = auto
system minRawFileSyncSecs = disable
system partialServiceMetaPeriod = 0
system quarantineFutureSecs = 2592000
system quarantinePastSecs = 77760000

[hartfoml]

system [firewall]
system assureUTF8 = false
system blockSignSize = 0
system blockSignatureDatabase = _blocksignature
system coldPath = $SPLUNK_DB/firewall/colddb
system coldToFrozenDir =
system coldToFrozenScript =
system compressRawdata = true
system defaultDatabase = main
system enableOnlineBucketRepair = true
system enableRealtimeSearch = true
system frozenTimePeriodInSecs = 48211200
system homePath = $SPLUNK_DB/firewall/db
system indexThreads = auto
system maxBloomBackfillBucketAge = 30d

[alacercogitatus]

What does the output of that command show? What version are you using?

[hartfoml]

/opt/splunk/etc/system/local/indexes.conf

Sorry typing too fast for my own good.