Why am I still seeing old data?

[Santosh2]

Splunk data retention period is for 7 days.

But i could still see 2 years back data now.

I am not sure why? 

Can anyone help on this 

[gcusello]

Hi @Santosh2,

retention is managed in Splunk at bucket level, so a bucket will be discarded when the earliest event of the bucket will exceed the retention period.

probably in your bucket you still have events that don't exceed the retention period.

You can check them using the Monitoring Console.

Ciao.

Giuseppe

[Santosh2]

i am new to splunk, can u please tell indetail, how can check in monitor console. And what will be the solution 

[gcusello]

Hi @Santosh2,

you can see in Monitoring Console at [Settings -- Monitoring Console -- Indexing -- Indexes and Volumes -- Indexes and Volumes:Instances]: there's the age (in days) of earliest and latest event in an index.

Clicking on an index, you drilldown in the details dashboard and you have all the information you need.

About solution, what do you mean?

you cannot change the situation of indexed data, you could modify index parameters (e.g. bucket dimension), fo the new buckets, but my hint is to leave the default, even if you have old data.

Maybe you could analyze your indexes structure: in other words, an index is created for events with the same retention and the same access grants, in addition it's a best practice put in the same index data with a similar volume of ingestion (e.g. don't put in the same index a data flow of many GB/day with a data flow of few events by day.

I say this because many new Splunk users think to Splunk as a DB and to an index as a DB table, infact they have one sourcetype for each index; Splunk isn't a DB and an index isn't a table.

As I said an index usually contains many sourcetypes with the same retention and the same grants.

Ciao.

Giuseppe

[ITWhisperer]

Data retention is based on buckets - perhaps the bucket holds both old and new(er) data so hasn't been archived yet?

[Santosh2]

sorry for the late reply....
Yes as you said the buckets contains old and new data, i checked my props.conf, everthing looks good and no warnings are seen in spunkd logs as well, linebreaking is happening correctly, but still i am able to see 2 years old data.
I tried to change the forzentimeperiod to 3  days as well, but still i can see the old data.

[gcusello]

Hi @Santosh2,

as I said, retention is managed in Splunk at bucket level, so a bucket will be discarded when the earliest event of the bucket will exceed the retention period.

probably in your bucket you still have events that don't exceed the retention period.

Ciao.

Giuseppe

[ITWhisperer]

I think @gcusello means the latest event in the bucket not the earliest, that is, the bucket will be kept until the most recent event in the bucket is older than the retention period. With very few events entering the bucket, this may take a while.