Reporting

Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

Communicator

Prior to updating to Splunk Enterprise 8.0.2 scheduled accelerated reports ran extremely fast:
Report A
Duration: 37.166
Record count: 314

After updating to Splunk Enterprise 8.0.2 the report ran extremely slow:
Report A
Duration: 418.621
Record count: 300

Given the patch notes for 8.0.2 – I'm not seeing any changes to acceleration or summary indexing, so is it safe to assume this is a fluke?

The massive increase in report generation (job) time of the scheduled accelerated reports appears to be caused by them no longer accessing the corresponding report acceleration summary. The "Access Count" never goes up when the scheduled reports are run.

alt text

Guess we'll wait for 8.0.3 to fix this.

Troubleshooting steps attempted:
Manually rebuild Report Acceleration Summaries
Delete all affected Report Acceleration Summaries
Delete and recreate affected production reports – recreated schedule and checked box for acceleration
Check filesystem permissions of inputlookup csv - confirmed -rw-rw-r-- splunk splunk

Labels (2)
0 Karma

Splunk Employee
Splunk Employee

It would be useful to know which version you were on before when it was working, and whether the inputlookup is done as a subsearch.

0 Karma

Communicator

Version 8.0.1 was used prior to the issue starting after upgrading to 8.0.2. The inputlookup function used is [| inputlookup filename.csv | fields name1 | rename name1 as my_search_value | format] and comes after index=index_name in the search query.

0 Karma

Communicator

Downgrading to 8.0.1 resulted in acceleration working correctly again. Do you know if there's a full list of changes in 8.0.2 that we can review for a possible cause?

0 Karma

SplunkTrust
SplunkTrust

Did you log a support case? As that way the support team will actually confirm that its a bug/fix the issue in a future release...

0 Karma

Communicator

"It appears you do not have an active Support Contract or entitlement and as a result, cannot open a Support case. If you believe this is an error, please contact 1-855-SPLUNK S, or consult https://www.splunk.com/en_us/about-us/contact.html#customer-support for a country specific Support phone number and we can resolve any contractual data integrity issues."

0 Karma

SplunkTrust
SplunkTrust

Oh ok, if your not on a customer site you cannot raise an issue, if i get spare time I'll try it 🙂

0 Karma

Communicator

Were you able to report a support case? Has anyone else reported this issue?

0 Karma

SplunkTrust
SplunkTrust

Will try and replicate it today...where does the summarisation load screen come from?

0 Karma

Communicator
0 Karma

SplunkTrust
SplunkTrust

So I did this, search 1:

index=_internal | stats count by _time

Search 2:

index=_internal | stats count by _time | inputlookup append=true rest_api_test.csv

Report acceleration confirms they will both use the same acceleration job.
When I run the reports they advise they both used the same acceleration job.

So did not replicate it, is that similar to what you are doing?

0 Karma

SplunkTrust
SplunkTrust

Search 2 is scheduled and working as expected, access count has increased to 6 now (did not replicate your issue)!
Splunk 8.0.2

0 Karma

Communicator

Try doing the inputlookup as a subsearch. In my case, it looks like this:

 [| inputlookup file.csv | fields title | rename title as TITLE | format]
0 Karma

SplunkTrust
SplunkTrust

Replicated as a subsearch via:
index=_internal | stats count by _time | append [ inputlookup rest_api_test.csv ]

0 Karma

Communicator

Thanks, I'll update OP to indicate subsearch requirement.

0 Karma

SplunkTrust
SplunkTrust

Support confirmed "According to the search.log, there is a change in the behavior of the search using append between 8.0.1 and 8.0.2." investigation will now go to sustaining...

0 Karma

Communicator

Fantastic, thanks for submitting the case!

0 Karma

Communicator

Schedule the second search you created to run at some interval and see if the "Access Count" number increases after each scheduled report runs. In my case, the count doesn't go up and the report generates slowly – despite showing that it is using the acceleration (summary) index on the Report Acceleration Summaries page.

0 Karma

Splunk Employee
Splunk Employee

And is the summary showing as complete? Perhaps the summary is not in a usable state yet?

0 Karma

Communicator

Yes, the summary was 100% complete as shown in OP screenshot. Prior to the 8.0.2 upgrade the report acceleration was working as intended.

0 Karma

Communicator

It could, that would be greatly appreciated. Thanks!

0 Karma

SplunkTrust
SplunkTrust

Sorry I do not have good news, basically embedded subsearches + newer Splunk versions + report acceleration do not work together, I believe it relates to fixing a previous issue...

The fix will come in a future version but no ETA so I cannot help further!

The only solution is to not have embedded sub searches....or don't accelerate 🙂

0 Karma