Getting Data In

how can I check 7 years of old data from splunk?

prasireddy
Explorer

Hi Team,
how can I check 7 years old data that means the first ingestion was on 26 dec of 2016 I need total data size from starting date to Jun 30 2023.
I have tried with following Query ,when I run that its showing some 
1."DAG Execution Exception Error ":search has cancelled 
2.search Auto-cancelled 

the Query which I have used 

index=wineventlog source=security command_type!="METER_ALERT"
|eval size=len(_raw)
| eval raw_len_KB= round(size/1024,3)
| eval raw_len_MB = round(size/1024/1024,3)
| eval raw_len_GB = round(size/1024/1024/1024,3)
| table size,raw_len_KB,raw_len_MB ,raw_len_GB,index
| stats count sum(size) as Bytes sum(raw_len_KB) as KB sum(raw_len_MB) as MB sum(raw_len_GB) as GB by index


please help on this ?

Thanks In Advance 
Bala



Labels (5)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @prasireddy,

Did you checked in the indexes.conf where the index is defined?

To be more sure, you should check using the btool:

$SPLUNK_HOME/bin/splunk camd btools indexes list --debug

maybe the option in another indexes.conf.

In addition you could use the Monitoring Console to see the retention in your index and how old is the earliest event in your index.

Ciao.

Giuseppe

View solution in original post

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Even assuming that you indeed have the data (you can check it in various ways, from looking at your actual bucket directories to tstats and dbinspect) your search is very inefficient and will probably get auto-canceled due to resource exhaustion.

Also, instead of manually calculating those stats, you can get most of the same info from the dbinspect command.

0 Karma

prasireddy
Explorer

Hi @PickleRick ;
I have used dbinspect command  but here I need sourcetype and even the total count and size of command_type="METER_ALERT" and NON "METER_ALERT" separately .

Query :   index=service_audit sourcetype=SMWAN command_type !="METER_ALERT"
|eval size=len(_raw)
| eval raw_len_KB= size/1024
| eval raw_len_MB = size/1024/1024
| eval raw_len_GB = size/1024/1024/1024
| table size,raw_len_KB,raw_len_MB ,raw_len_GB,index
| stats count sum(size) as Bytes sum(raw_len_KB) as KB sum(raw_len_MB) as MB sum(raw_len_GB) as GB
by index

 

 

 

0 Karma

prasireddy
Explorer

Yes, For shorter periods it is working fine.
How can I check retention time grater than seven years 
and it is not a summary index .

Please could you help on this

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @prasireddy,

you should check in the indexes.conf file where the index is defined, what's the retentin period of the logs contained in that index.

Retention is define using the option "frozenTimePeriodInSecs".

If it isn't defined, you have the default six years retention period.

Ciao.

Giuseppe

0 Karma

prasireddy
Explorer

Hi ,
I have checked indexex.conf but here I did not find Retention option like  "frozenTimePeriodInSecs". then it means default is 6years .
Even when I'm giving 6 yr periods I did not see the data why ?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @prasireddy,

Did you checked in the indexes.conf where the index is defined?

To be more sure, you should check using the btool:

$SPLUNK_HOME/bin/splunk camd btools indexes list --debug

maybe the option in another indexes.conf.

In addition you could use the Monitoring Console to see the retention in your index and how old is the earliest event in your index.

Ciao.

Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @prasireddy,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

prasireddy
Explorer

I will check come back 

thank you so much 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @prasireddy,

i suppose that you have a retention time grater than seven years for your data modifying the default value (6 years), otherwise it isn't directly possible,  the only solution is storing aggegated results in a summary index (with a retention grather than 7 years) and then run searches on this summary index.

Anyway, have you results to your search using e shorter period?

Ciao.

Giuseppe

 

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...