topic Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches in Reporting

Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

orion44 — Mon, 08 Jun 2020 23:34:49 GMT

Prior to updating to Splunk Enterprise 8.0.2 scheduled accelerated reports ran extremely fast:
Report A
Duration: 37.166
Record count: 314

After updating to Splunk Enterprise 8.0.2 the report ran extremely slow:
Report A
Duration: 418.621
Record count: 300

Given the patch notes for 8.0.2 – I'm not seeing any changes to acceleration or summary indexing, so is it safe to assume this is a fluke?

The massive increase in report generation (job) time of the scheduled accelerated reports appears to be caused by them no longer accessing the corresponding report acceleration summary. The "Access Count" never goes up when the scheduled reports are run.

Guess we'll wait for 8.0.3 to fix this.

Troubleshooting steps attempted:
Manually rebuild Report Acceleration Summaries
Delete all affected Report Acceleration Summaries
Delete and recreate affected production reports – recreated schedule and checked box for acceleration
Check filesystem permissions of inputlookup csv - confirmed -rw-rw-r-- splunk splunk

Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

codebuilder — Fri, 14 Feb 2020 20:47:01 GMT

Did you migrate the scheduled report or recreate it?

Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

orion44 — Fri, 14 Feb 2020 21:21:35 GMT

Total delete and recreate the report, schedule, acceleration, and the corresponding report acceleration summary.

Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

orion44 — Sat, 15 Feb 2020 20:36:07 GMT

After further testing, it appears that reports that use inputlookup commands no longer accelerate correctly after the Splunk Enterprise 8.0.2 update. I reached this conclusion by removing the inputlookup command from the two reports mentioned above in my question – then hardcoded a few example values into my Splunk search query to test.

After doing so, the report acceleration performed as expected. Sadly this isn't a solution and only a workaround until the issue is (hopefully) fixed in future Splunk Enterprise releases.

CONFIRMED
Splunk 8.0.2 caused report acceleration to break. I was able to correct the issue by rolling back to version 8.0.1 via the steps below:

wget -O splunk-8.0.1-6db836e2fb9e-linux-2.6-x86_64.rpm 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.0.1&product=splunk&filename=splunk-8.0.1-6db836e2fb9e-linux-2.6-x86_64.rpm&wget=true'

sudo rpm -Uvh --oldpackage splunk-8.0.1-6db836e2fb9e-linux-2.6-x86_64.rpm

sudo service splunk start

Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

gjanders — Mon, 17 Feb 2020 06:11:50 GMT

Did you log a support case? As that way the support team will actually confirm that its a bug/fix the issue in a future release...

Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

orion44 — Mon, 17 Feb 2020 17:55:23 GMT

"It appears you do not have an active Support Contract or entitlement and as a result, cannot open a Support case. If you believe this is an error, please contact 1-855-SPLUNK S, or consult https://www.splunk.com/en_us/about-us/contact.html#customer-support for a country specific Support phone number and we can resolve any contractual data integrity issues."

Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

gjanders — Mon, 17 Feb 2020 22:26:13 GMT

Oh ok, if your not on a customer site you cannot raise an issue, if i get spare time I'll try it 🙂

Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

orion44 — Tue, 18 Feb 2020 03:12:57 GMT

It could, that would be greatly appreciated. Thanks!

Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

martin_mueller — Wed, 19 Feb 2020 10:14:36 GMT

What's the search?

Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

orion44 — Wed, 19 Feb 2020 20:22:48 GMT

Outside of the inputlookup command the search itself is irrelevant. Once I remove the reference to inputlookup command from the search and hardcode a few examples from the csv – the search accelerates as expected.

Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

nvanderwalt_spl — Thu, 20 Feb 2020 00:18:36 GMT

It would be useful to know which version you were on before when it was working, and whether the inputlookup is done as a subsearch.

Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

orion44 — Thu, 20 Feb 2020 08:22:37 GMT

Version 8.0.1 was used prior to the issue starting after upgrading to 8.0.2. The inputlookup function used is [| inputlookup filename.csv | fields name1 | rename name1 as my_search_value | format] and comes after index=index_name in the search query.

Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

orion44 — Fri, 21 Feb 2020 23:17:41 GMT

Downgrading to 8.0.1 resulted in acceleration working correctly again. Do you know if there's a full list of changes in 8.0.2 that we can review for a possible cause?

Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

orion44 — Wed, 26 Feb 2020 18:56:17 GMT

Were you able to report a support case? Has anyone else reported this issue?

Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

gjanders — Mon, 02 Mar 2020 22:33:13 GMT

Will try and replicate it today...where does the summarisation load screen come from?

Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

orion44 — Tue, 03 Mar 2020 00:05:14 GMT

Thanks!

It can be found at https://:8000/en-US/manager/system/summarization

Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

gjanders — Tue, 03 Mar 2020 01:19:11 GMT

So I did this, search 1:

index=_internal | stats count by _time

Search 2:

index=_internal | stats count by _time | inputlookup append=true rest_api_test.csv

Report acceleration confirms they will both use the same acceleration job.
When I run the reports they advise they both used the same acceleration job.

So did not replicate it, is that similar to what you are doing?

Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

orion44 — Tue, 03 Mar 2020 01:28:30 GMT

Schedule the second search you created to run at some interval and see if the "Access Count" number increases after each scheduled report runs. In my case, the count doesn't go up and the report generates slowly – despite showing that it is using the acceleration (summary) index on the Report Acceleration Summaries page.

Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

nvanderwalt_spl — Tue, 03 Mar 2020 02:32:07 GMT

And is the summary showing as complete? Perhaps the summary is not in a usable state yet?

Re: Splunk 8.0.2 report acceleration broken for reports using inputlookup command in subsearches

orion44 — Tue, 03 Mar 2020 02:47:42 GMT

Yes, the summary was 100% complete as shown in OP screenshot. Prior to the 8.0.2 upgrade the report acceleration was working as intended.