Why are my dashboard panels giving an error when being created using report reference as base search?

[mbasharat]

Hi,

I have a simple scheduled report.

This report is scheduled and runs over night every day and gets me data for 1 year. It takes about an hour for this report to run over night so I can use it in the morning. This report has about 22 fields populated. The number of statistics results it populates is about 3 million rows. This report is accelerated for 1 year.

The next morning, when I come and look at completed jobs, it is always successful and it opens up perfectly fine.

Now, the trouble I am having is I am using this report to create dashboard panels using a report as a reference in a base search. When I try to get for example, simple stats(dc), it throws me the following error: "Error while fetching data" after few seconds and fails to populate my dashboard panels. However, when I run the same report for a smaller time span, e.g. 2 months or 3 months, then the dashboard panels load fine. They do take some time though but they load as I expect them to.

Why am I getting an error for a 1 year report when populating dashboard panel vs 2/3 months version of report? Maybe dashboard panel has some kind of rendering restrictions etc...??

It is a simple report, and when I simply pull the report the next morning, it literally takes about 3 seconds for it to show me all 3 million stats it pulled last night for the whole year. But, the problem is when I use it to create dashboard panels and also, even for smaller report for 2/3 months, it takes some time but it loads.

Larger span like a 1 year, it fails as stated above.

I am just trying to create several panels based off of one report for optimization and performance perspective, and I have been successful in pretty much all my work using this strategy. It is just this one that is giving me a hard time — possibly because it is a larger report? How do I fix this issue? And also, how do I get the panels to populate quick along with it?

Thanks in advance for the guidance.

[martin_mueller · Accepted solution]

By default a base search can only be 500k results: http://docs.splunk.com/Documentation/Splunk/7.2.0/Viz/Savedsearches#Post-process_searches_2

You can either try increasing the setting - no idea if 10x the default will work well - or try to reduce the results count of your base search. How to do the latter best depends on your use case.

[martin_mueller]

Any app you use for keeping search head configuration in, local or - given that it's your app - default, limits.conf.

[mbasharat]

Actually, we have designated apps for everything. This particular report is also in its specific App named ABC. What is the location of the file for this setting in any app?

Thanks

[martin_mueller]

It's a searchhead setting. Ideally you'll create an app that contains this setting, deploy it to your dev searchhead, see if it works, and then deploy it to your SHC.

In almost all use cases it's possible to define smaller datacubes that power your panels... but again there's no way to help you there without knowing more details about your searches and the use cases you're trying to power with them.

[mbasharat]

My development searchhead is not in a cluster, so to change the max_count settings in limits.conf @ search head? And where exactly? The one in /server or /local?

What about the clustered area? Does it need to be changed on each searchhead in cluster if it works?

Reducing the results count is not an option for me because I need all results as the report is for vulnerabilities for each asset.

Again, I am trying to reference report as base search for a dashboard panel where I am running stats to get results FROM the saved report.

I think increasing limit should work since dashboard panels populate when I use a smaller reprt e.g. 1month-3months.

Thanks and awaiting response.