Splunk Search

Why past data is missing even if date range is inside my retention policy of that index?

muizash
Path Finder

SPL:
"(index=3y OR index=3mon) (host=x OR host=y)
name="RegisteredUserLog" actionType=egg pointGet=true (platform=0 OR platform=1)
| eval earned_date=strftime(_time, "%Y-%m-%d")
| stats count by event_id earned_date
| rename event_id as easy_id
| table easy_id earned_date"

Notes
- The data I am seeing today is different from when i saw and exported same data before 1 moth providing same date range.
- To give you idea, I am seeing 20K less results as compared to 1L events before one month for exact SPL and exact time range.
- Retention of index is not issue
- Date range is not issue

Please help
Thanks

0 Karma

gcusello
Legend

Hi @muizash,
did you already checked the max size of your index? if you reached it, the oldest buckets were deleted.

Ciao.
Giuseppe

0 Karma

muizash
Path Finder

Hi @gcusello
Even if maxTotalDataSizeMB is reached and events were frozen. We have retention of 3years. Why would a event of Jan 2020 be frozen? Or freezing is random? Isn't freezing based on age? Oldest events will be frozen even if maxTotalDataSizeMB is reached?

0 Karma

gcusello
Legend

Hi @muizash,
the oldest buckets are frozen when one of the two conditions is reached, max size or retention period.
Anyway it's strange that events of January 2020 are frozen, have you older events or not?

Ciao.
Giuseppe

0 Karma

muizash
Path Finder

Hi @gcusello
Yes, I am definitely able to see older data.
I wonder why some data is missing.
Is there any other possibility?

Thanks
Muiz

0 Karma

gcusello
Legend

Hi @muizash,
if you didn't deleted any event the problem could be in the search you're using:
could the missing events have timestamp between january 2 and january 12?
In other words, is the time format of your data dd/mm/yyyy?
In tis case there could be a parsing error.
Try to run a search using always as time range (eventually blocking search with head 10000) and see if there are future events caused by an error in timestamp.

Ciao.
Giuseppe

0 Karma

muizash
Path Finder

Thanks for suggestion @gcusello
Everything looks fine by that also.
Still cant solve the issue.

0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

Check if data is deleted because of retention or max size in last 1 month.

 index=_internal sourcetype=splunkd INDEX_NAME component=BucketMover frozenTimePeriodInSecs OR maxTotalDataSizeMB
0 Karma

muizash
Path Finder

Hi @manjunathmeti
Retention is 3 years are events from Jan 2020 are missing.
Even if maxTotalDataSizeMB is reached and events were frozen. We have retention of 3years. Why would a event of Jan 2020 be frozen? Or freezing is random? Isn't freezing based on age? Oldest events will be frozen even if maxTotalDataSizeMB is reached?

0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

Hi @muizash,
Refer indexes.conf documentation, this says:

maxTotalDataSizeMB = <>
** CAUTION: This setting takes precedence over other settings like 'frozenTimePeriodInSecs' with regard to data retention. If the index grows beyond 'maxTotalDataSizeMB' megabytes before 'frozenTimePeriodInSecs' seconds have passed, data could prematurely roll to frozen. As the default policy for rolling data to frozen is deletion, unintended data loss could occur.**

So data will be deleted if maxTotalDataSizeMB is reached even though events are not older than frozenTimePeriodInSecs.

0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!

Review:





Or Learn More in Our Blog >>