Hello all,
I've run into a problem with the backfill upon creating (also tried cloning) a KPI in regards to Splunk License Metrics using the following search:
index=_internal source=*license_usage.log type="Usage"
| fields idx, b
| eval indexname = if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx)
| bin span=5min _time
| stats sum(b) as b by indexname, _time
| eval GB=round(b/1024/1024/1024, 3)
| fields _time, indexname, GB
The Use Case:
I want a KPI for the License Usage with the separate Indexes as Entities.
Configuration info:
Seeing as I want the License Info on an per Index-Basis I konfigured the KPI to be split into Entities by the field "indexname".
As for the Frequency and Calculation I selected:
Calculating Maximum of GB per entity as entity value,
Sum of entity value as aggregate over the last 5 minute(s) every 5 minute(s).
Fill gaps in data with Null values and use a unknown threshold level for them.
So far so good... now I also configured a Backfill for the last 30 days (taxing on the system but it should manage).
The Problem:
Upon seeing the Message that the backfill was completed, I checked the itsi_summary Index and found the backfill data of the KPI but with regular gaps. More precisely, for each day it had backfilled the data from the activation time of the kpi (here 12:30) for about 6h (18:25/18:30) and then there were no further values for the day until the next day around 12:30. Even though there is license usage during the gap times and also available in the license_usage.log used by the KPI search.
The Data since activation is continuous and has no gaps.
I tried cloning the KPI, remaking the KPI with both adhoc or base search, but all featured the same curious results (just with different starting points as the activation time of the KPI was different).
Thus now I am wondering if there is some sort of limit for backfilling or if perhaps someone has an idea what caused this strange backfill behaviour? (Also there was no error message in the _internal index as far as I could tell.)
Help and ideas would be appreciated. Thanks in advance.
I also tried just with 7 days Backfill, but sadly got the same results.
7 days is also the minimum Splunk ITSI offers as a backfill option.
If you write backfill for 1 day, do you mean to manually fill the itsi_summary index? As in not going via the GUI of Splunk ITSI?
