Splunk Search

"An error occurred while fetching data" using a base search in a SH cluster

dkoops
Path Finder

Here is the case:

I've build a dashboard with 6 graphs/tables all using the same base search. It works like a charm on a single SH (where I've developed the app). After pushing it to our SH cluster, however, the dashboard crashes when loading more than a few days data, with the error "An error occurred while fetching data" in several or all graphs (it seems kind of random).

It starts loading normally, but then halfway "hangs" and shows the error mentioned above. It only does this when loading > 6 days data, which is (only) 46.555 events with a bundle size of 4,2 MB. With slightly less events all works well. The search log shows no errors at all (it even reports a successful completion), and only one warning at the start of the search.log that seems unrelated:

WARN  DistributedInfoSingleton - Failed to read symptoms of peer=SH02

The splunkd.log also shows nothing related.

Anyone an idea what's wrong? Or a tip on where to start troubleshooting?

0 Karma
1 Solution

tdime
Explorer

I had a similar issue with the same error message and found it to be related to the length of the postprocess search. You can test by moving more of the search into the base search or creating a macro to hold the postprocess search contents. Good luck.

View solution in original post

tdime
Explorer

I had a similar issue with the same error message and found it to be related to the length of the postprocess search. You can test by moving more of the search into the base search or creating a macro to hold the postprocess search contents. Good luck.

jrballesteros05
Communicator

Hello, in my case is a bit different, I only have the problem with a non-admin user with a custom role. If I run my dashboards as admin it runs perfectly.

Do you know what is going on there?

0 Karma

tmurata_splunk
Splunk Employee
Splunk Employee

maybe the disk quota if you see the error for specific user roles.

0 Karma

tdime
Explorer

are you sure the user has access to the indexes or can you view the search run which converts the tokens to values and copy/paste that in another splunk search running as your custom user?

0 Karma

jrballesteros05
Communicator

I also voted for your answer because it helped me when I was building the dashboards. Thank you very much.

0 Karma

jrballesteros05
Communicator

Hi, I am completely sure. I was trying with Opera browser and I had the problem I mentioned before but when I try with Firefox and Chrome it works perfectly. I read the Splunk doc and I've just realised that Opera is not listed in the supported browsers by Splunk.

http://docs.splunk.com/Documentation/Splunk/6.5.1/Installation/Systemrequirements

dkoops
Path Finder

It has been a while since I reported the issue, but it indeed seemed to have something to do with the base search. After moving some query stuff around in the dashboard xml it worken again at some point (unfortunately I'm unable to recall exactly what I did).

Another detail is that I was also using Opera browser at that point. Never occured to me that the browser might have been the issue, but it's a good thing to keep in mind for possible future troubleshooting. I cannot validate this anymore, but it might be helpful to add.

I will close this question, thx all.

0 Karma
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...