topic Re: ITSI - backfill is not working - past data is unavailable in Splunk ITSI

ITSI - backfill is not working - past data is unavailable

andrewpagans — Wed, 09 May 2018 13:36:44 GMT

Hi Splunkers,

I have an issue. I created a KPI and enabled backfill for 7 days.
The problem is that in Glass Table / Deep Dive the data is visible from the activation of KPI.
I am not able to see for example Yesterday's data.

Can you help me?

Last info : I have 2/3 months of data in the related index.

Thanks a lot

Re: ITSI - backfill is not working - past data is unavailable

skoelpin — Wed, 09 May 2018 14:05:56 GMT

Did you wait for the backfill to complete? You should see a message popup once the backfill is done, if that message hasn't appeared yet then you must wait until its done and you will see your data

Re: ITSI - backfill is not working - past data is unavailable

andrewpagans — Wed, 09 May 2018 14:14:32 GMT

Hi , thanks for reply.

Yes, I saw the message (data backfilled), it appeared after 5 minutes.
I checked again after 4/5 hours, but data it was unavailable.

For Troubleshooting, Do you have any suggestions?

Thanks again

Re: ITSI - backfill is not working - past data is unavailable

skoelpin — Wed, 09 May 2018 14:21:13 GMT

You should look in the ITSI summary index to see if the data is present there. If the backfill did not work correctly, then you will not see data here thus not giving you values in the glasstable view.

Go to your ITSI search head and type in index=itsi_summary and look for your KPIs that you backfilled and build a simple timechart to see how far back it goes

You could also try to build another test service and add a few KPI's and try to backfill again to verify if it works. Also what version of ITSI are you running?

Re: ITSI - backfill is not working - past data is unavailable

andrewpagans — Wed, 09 May 2018 14:27:53 GMT

Hi,
Yes I did exactly this, I created a new TEST Service with this KPI and it doesn't work.
Ok, I'll try to look into index=itsi_summary, and I will let you know.
ITSI version is 3.0.2

Thank you!

Re: ITSI - backfill is not working - past data is unavailable

skoelpin — Wed, 09 May 2018 14:42:18 GMT

Did you duplicate the KPIs to the new service or did you re-create them from scratch? You will need to re-create them from scratch and re-apply the backfill on that new service.

Are the values showing in the deep-dive view and not the glasstable view?

Re: ITSI - backfill is not working - past data is unavailable

skoelpin — Wed, 09 May 2018 14:49:39 GMT

Also, I had a similar issue with ITSI, but I kept getting N/A's for all glasstable values that were older than 23 days. I filed a ticket with support and they fixed it in the newest version (see below)

2018-04-11 ITOA-11031 After daylight savings, glass tables do not populate with End Date and Time beyond 3 weeks. The issue persists for 30 days after the time change.

http://docs.splunk.com/Documentation/ITSI/latest/ReleaseNotes/Fixedissues

Re: ITSI - backfill is not working - past data is unavailable

andrewpagans — Wed, 09 May 2018 15:34:19 GMT

Really thanks for your support.

I tried all the combination in a new test service:
- KPI cloned
- KPI created from scratch , using ad hoc search
- KPI created from scratch , using KPI Base Search

In deep dive ( last 5 hours) :
- I have no data previous KPI activation.
- Comparing with Yesterday, there are completely no data.

In index=itsi_summary there are really few events in the previous days.
I was expecting a lot of events.

I am also checking Skipped searches.

Will try with new version 🙂

Re: ITSI - backfill is not working - past data is unavailable

skoelpin — Wed, 09 May 2018 16:06:46 GMT

The glasstable view shows values based off a single point in time, where the deep-dive view shows values over a range of times. So for example, if you have 1 minute windows set, and you did not get data within that 1 minute window, it will show N/A in the glasstable view. Whereas, it will show in the deep-dive views since the windows are relative to what timerange you set.

So it sounds like your backfill is not working correctly since you claimed to have backfilled the past 7 days but there are few data points. You should first run a query over your raw data to verify you have data to summarize in your summary index. You can also try to create a test service and KPI against your internal index which has a steady stream of data. You should then backfill this into your new KPI for the past 7 days to see if you get values. If this produces N/A's then you definitely have a problem and need to open a support case