Getting Data In
Highlighted

Automatically source is getting deleted after 24 hours

Explorer

I added source file (.csv file) to splunk using below command,

./splunk add oneshot /root/project/2003.csv –sourcetype sfpd

I can see that 1,50,902 events got indexed.

But exactly after one day, all indexed data from this source file will get deleted except one line (i.e., header of .csv).

I haven't executed delete command. Also I removed the privileges of using delete command, so no one can use it. But still this issue is happening daily.

I am not able to find the solution for this issue.

Please someone help me. Thanks for your help.

Tags (3)
0 Karma
Highlighted

Re: Automatically source is getting deleted after 24 hours

SplunkTrust
SplunkTrust

A good idea will be to check the splunk data retention period for the index where this source's data is stored. Indexer.conf-> FrozenTimePeriodInSecs attribute. If this attribute exists for your index and its value is 86400, this is the problem. Increase the value to required period in second, and restart the splunk instance.

0 Karma
Highlighted

Re: Automatically source is getting deleted after 24 hours

Super Champion

What somesoni2 said, and when the data is searchable check the timestamp of the data:
search yourdata | table _time,_raw
The _time value should match the time in the _raw string, and both should make sense.

0 Karma
Highlighted

Re: Automatically source is getting deleted after 24 hours

Explorer

I checked indexes.conf -> FrozenTimePeriodInSecs attribute. Its value is 188697600.

I also ran the following command - "search yourdata | table time,raw" as you suggested. The _time value matched with the time in _raw string. Time stamp for the data is 2003-12-01. As the data is 10 years old, may be data is getting deleted. Is it is true? then in that case how I can resolve this issue. Please let me know

Highlighted

Re: Automatically source is getting deleted after 24 hours

Explorer

The following attribute - maxHotIdleSecs in Indexes.conf file has the value 86400. Is this is the reason for this issue??

0 Karma
Highlighted

Re: Automatically source is getting deleted after 24 hours

Ultra Champion

You've identified the problem:

"the data is 10 years old".

The default retention period that you see in frozenTimePeriodInSecs is about 6 years. That means that as soon as splunk gets time time make the comparison, which in your case is when the hot bucket rolls to warm, it will correctly see that the data should be deleted, and does so.

The solution is to increase the value for frozenTimePeriodInSecs to a higher value, e.g. 400000000 or 500000000, which is about 12 and 15 years, respectively. The highest possible value is 4294967295, which is more than a hundred years...

You can read more about data retention here:

http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Setaretirementandarchivingpolicy

Hope this helps,

/K

View solution in original post

Highlighted

Re: Automatically source is getting deleted after 24 hours

Explorer

I modified the frozenTimePeriodInSecs to set 400000000 as its value. If I face this issue again, I will message here. Also I want to let you know that, I modified "maxHotIdleSecs" value from 86400 to 604800. Thanking everyone.

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.