Retirement and archiving period for one file

[sushildabare]

We have a file which will be updated very rarely(may be once a year or so that too may be a line will be added or deleted).
We have used sourcetype = String crcSalt = String for this file in monitor input stanza.
Without crcSalt Indexer is not reading this file.

We have set retirement and archiving policy as 3 months for all splunk data. So after 3 months will splunk automatically read the above file with crcSalt?
Can we modify the archiving period for only one source/file? say never archive, only for this one source.

Thanks|

[sdwilkerson · Accepted solution]

Sushildabare,

I think you may be confusing inputs.conf with indexes.conf. Inputs.conf handles all inputs (as you mentioned); however, only indexes.conf adjust your indexer's retention policies. So, if Splunk retires data at 3 months there is nothing about that config that would tell it to pull the data in again since that is handled separately.

Based on what you said here, all data will be retired when it is 3 months old. If the file in question has not yet changed, Splunk will not know to re-index it automatically.

Essentially, your policy told Splunk to purge the file.

You can adjust retention policies by index, but not by index-contents. If you want data in this file to be preserved in Splunk for a different period of time than other data, then I suggest you input the contents of this file into a separate index that has its own retention period.

Depending on what you get out of this file, you also might want to consider pulling it in as a lookup table (not normal indexed data). Maybe you have a scripted input run once a day, pull this data in to Splunk, then have a search which dumps the data out to your lookup. This would require a lot of re-indexing of this data, but if it is small this is trivial for both the system work and your license.

Another idea is to use the splunk-base app getwatchlist if this file is accessible via ftp/http/https and has any kind of standard delimiter. This would allow you to run a regular search in Splunk (e.g. daily) to pull in this list to the search and dump it directly out to a lookup and not have to index the data at all.

Best
Sean

[sushildabare]

Thanks a lot for your help..

[sdwilkerson]

BTW, if all of this was helpful, you can "accept" my answer and click the up-arrow to "up vote" my answer.

[sdwilkerson]

Sushildabare,
Yes, this looks like it should work for what you want to achieve.

[sushildabare]

[default]
frozenTimePeriodInSecs = 7776000
[XYZ]
frozenTimePeriodInSecs = 188697600
[erp]
coldPath = $SPLUNK_DB\erp\colddb
[erp_maxdb]
coldPath = $SPLUNK_DB\erp_maxdb\colddb
etc etc.... for remaining indexes.

I have used 3 months in [default] stanza which will be global setting.Will this keep data of all indexes for 3 months and XYZ index data for 6 years? Please confirm.

[sushildabare]

We have decided to go for creating a seperate index(say XYZ) for these files. Now we have around 10 indexes for which we will modify the indexes.conf file in the path D:\Program Files\Splunk\etc\apps\search\local
For 9 indexes we need to set frozenTimePeriod 3 months and only for this new index we want to keep it for 6 years so will the below indexes.conf file works fine ?

[sushildabare]

Hi Sean,
Thanks again! To create summary index I need to follow the steps mentioned in http://docs.splunk.com/Documentation/Splunk/4.2.4/Knowledge/Usesummaryindexing
section, "Setting up Summary index searches in splunk web" right ?
Actually we need the entire file content not only changes..
I will send you data sample of one of the file tomorrow as I cannot access my server now.

Thanks~

[sdwilkerson]

Sushildabare,
If the data is purged from the index, you will not be able to retrieve the information.
If you don't want to try one of the options I provided earlier, you could also write a separate search to summarize the target events and dump them to a summary index and configure Splunk to retain that info longer.
If you have configured Splunk to monitor the target file, then it should index any changes to that file but NOT necessarily reindex the entire file.
It sounds like what you want might be a lookup table and not indexed-data.
Submit a data sample if you want.

[sushildabare]

Hi Sean,
Thank you for your detailed explaination.
After 3 months, if the file in question has not yet changed then splunk will not reindex it automatically, I agree to this. Kindly clarify on below..
1. After 3 months can we get search results for these files(purged data, which is not reindexed) as we used to get before 3 months? I think search will not give any results, But please confirm on this.
2. Suppose the file is changed after 6 months will splunk read it and reindex it automatically?

Thanks~