Splunk Search

Why am I still seeing old data?

Santosh2
Path Finder

Splunk data retention period is for 7 days.

But i could still see 2 years back data now.

I am not sure why? 

Can anyone help on this 

Labels (1)
Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Santosh2,

retention is managed in Splunk at bucket level, so a bucket will be discarded when the earliest event of the bucket will exceed the retention period.

probably in your bucket you still have events that don't exceed the retention period.

You can check them using the Monitoring Console.

Ciao.

Giuseppe

0 Karma

Santosh2
Path Finder

i am new to splunk, can u please tell indetail, how can check in monitor console. And what will be the solution 

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Santosh2,

you can see in Monitoring Console at [Settings -- Monitoring Console -- Indexing -- Indexes and Volumes -- Indexes and Volumes:Instances]: there's the age (in days) of earliest and latest event in an index.

Clicking on an index, you drilldown in the details dashboard and you have all the information you need.

About solution, what do you mean?

you cannot change the situation of indexed data, you could modify index parameters (e.g. bucket dimension), fo the new buckets, but my hint is to leave the default, even if you have old data.

Maybe you could analyze your indexes structure: in other words, an index is created for events with the same retention and the same access grants, in addition it's a best practice put in the same index data with a similar volume of ingestion (e.g. don't put in the same index a data flow of many GB/day with a data flow of few events by day.

I say this because many new Splunk users think to Splunk as a DB and to an index as a DB table, infact they have one sourcetype for each index; Splunk isn't a DB and an index isn't a table.

As I said an index usually contains many sourcetypes with the same retention and the same grants.

Ciao.

Giuseppe

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Data retention is based on buckets - perhaps the bucket holds both old and new(er) data so hasn't been archived yet?

0 Karma

Santosh2
Path Finder

sorry for the late reply....
Yes as you said the buckets contains old and new data, i checked my props.conf, everthing looks good and no warnings are seen in spunkd logs as well, linebreaking is happening correctly, but still i am able to see 2 years old data.
I tried to change the forzentimeperiod to 3  days as well, but still i can see the old data.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Santosh2,

as I said, retention is managed in Splunk at bucket level, so a bucket will be discarded when the earliest event of the bucket will exceed the retention period.

probably in your bucket you still have events that don't exceed the retention period.

Ciao.

Giuseppe

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

I think @gcusello means the latest event in the bucket not the earliest, that is, the bucket will be kept until the most recent event in the bucket is older than the retention period. With very few events entering the bucket, this may take a while.

0 Karma
Get Updates on the Splunk Community!

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...