topic Re: Why past data is missing even if date range is inside my retention policy of that index? in Splunk Search

Why past data is missing even if date range is inside my retention policy of that index?

muizash — Wed, 30 Sep 2020 04:31:10 GMT

SPL:
"(index=3y OR index=3mon) (host=x OR host=y)
name="RegisteredUserLog" actionType=egg pointGet=true (platform=0 OR platform=1)
| eval earned_date=strftime(_time, "%Y-%m-%d")
| stats count by event_id earned_date
| rename event_id as easy_id
| table easy_id earned_date"

Notes
- The data I am seeing today is different from when i saw and exported same data before 1 moth providing same date range.
- To give you idea, I am seeing 20K less results as compared to 1L events before one month for exact SPL and exact time range.
- Retention of index is not issue
- Date range is not issue

Please help
Thanks

Re: Why past data is missing even if date range is inside my retention policy of that index?

manjunathmeti — Mon, 09 Mar 2020 07:01:15 GMT

Check if data is deleted because of retention or max size in last 1 month.

 index=_internal sourcetype=splunkd INDEX_NAME component=BucketMover frozenTimePeriodInSecs OR maxTotalDataSizeMB

Re: Why past data is missing even if date range is inside my retention policy of that index?

gcusello — Mon, 09 Mar 2020 07:03:19 GMT

Hi @muizash,
did you already checked the max size of your index? if you reached it, the oldest buckets were deleted.

Ciao.
Giuseppe

Re: Why past data is missing even if date range is inside my retention policy of that index?

muizash — Mon, 09 Mar 2020 07:19:02 GMT

Hi @gcusello
Even if maxTotalDataSizeMB is reached and events were frozen. We have retention of 3years. Why would a event of Jan 2020 be frozen? Or freezing is random? Isn't freezing based on age? Oldest events will be frozen even if maxTotalDataSizeMB is reached?

Re: Why past data is missing even if date range is inside my retention policy of that index?

muizash — Mon, 09 Mar 2020 07:20:17 GMT

Hi @manjunathmeti
Retention is 3 years are events from Jan 2020 are missing.
Even if maxTotalDataSizeMB is reached and events were frozen. We have retention of 3years. Why would a event of Jan 2020 be frozen? Or freezing is random? Isn't freezing based on age? Oldest events will be frozen even if maxTotalDataSizeMB is reached?

Re: Why past data is missing even if date range is inside my retention policy of that index?

gcusello — Mon, 09 Mar 2020 07:28:04 GMT

Hi @muizash,
the oldest buckets are frozen when one of the two conditions is reached, max size or retention period.
Anyway it's strange that events of January 2020 are frozen, have you older events or not?

Ciao.
Giuseppe

Re: Why past data is missing even if date range is inside my retention policy of that index?

muizash — Mon, 09 Mar 2020 07:34:40 GMT

Hi @gcusello
Yes, I am definitely able to see older data.
I wonder why some data is missing.
Is there any other possibility?

Thanks
Muiz

Re: Why past data is missing even if date range is inside my retention policy of that index?

gcusello — Mon, 09 Mar 2020 07:51:51 GMT

Hi @muizash,
if you didn't deleted any event the problem could be in the search you're using:
could the missing events have timestamp between january 2 and january 12?
In other words, is the time format of your data dd/mm/yyyy?
In tis case there could be a parsing error.
Try to run a search using always as time range (eventually blocking search with head 10000) and see if there are future events caused by an error in timestamp.

Ciao.
Giuseppe

Re: Why past data is missing even if date range is inside my retention policy of that index?

manjunathmeti — Mon, 09 Mar 2020 07:52:29 GMT

Hi @muizash,
Refer indexes.conf documentation, this says:

maxTotalDataSizeMB = <>
** CAUTION: This setting takes precedence over other settings like 'frozenTimePeriodInSecs' with regard to data retention. If the index grows beyond 'maxTotalDataSizeMB' megabytes before 'frozenTimePeriodInSecs' seconds have passed, data could prematurely roll to frozen. As the default policy for rolling data to frozen is deletion, unintended data loss could occur.**

So data will be deleted if maxTotalDataSizeMB is reached even though events are not older than frozenTimePeriodInSecs.

Re: Why past data is missing even if date range is inside my retention policy of that index?

muizash — Mon, 09 Mar 2020 08:57:08 GMT

Thanks for suggestion @gcusello
Everything looks fine by that also.
Still cant solve the issue.