Dashboards & Visualizations

The Read Operation Timed Out - how do i stop this occurring?!

pj
Contributor

I am developing a dashboard that leverages the HiddenSavedSearch module to bring in saved search results set that contains about 250k events.

I am then using this search to populate a number of panels in the dashboard, where i am postprocessing the events to display summary tables, pie charts etc. I then allow the user to drill down to the results set.

The problem seems to be that being a big chunk of data to postprocess, it is taking possibly longer than Splunk will permit. As a result all my panels are displaying:

Results Error: 500 - Splunkd daemon is not responding: ('The read operation timed out',)

So, what would be the workaround here? I assume there might be some tweak i can make to extend the timeout period? On the odd occasion that the postprocessing completes within whatever the limit is, the dashboard renders fine. I cannot reduce the number of events as we have a lot of data to report on. Bringing the results back in this fashion using a hiddensavedsearch is a lot faster than actually searching for them directly.

We are using Splunk 4.3.2

Thanks!

1 Solution

pj
Contributor

Leverage summary indexing, report acceleration or data model acceleration. The new features of Splunk 6 should mean that there are other ways to render large amounts of data extremely quickly and in a manner that scales.

View solution in original post

0 Karma

pj
Contributor

Leverage summary indexing, report acceleration or data model acceleration. The new features of Splunk 6 should mean that there are other ways to render large amounts of data extremely quickly and in a manner that scales.

0 Karma

jluste
Path Finder

I'm using 5.0.8 and I have the same issues for certain post processed panels on a dashboard. The dashboard uses 1 saved (accelerated) search for like 15 panels. 1 or 2 of the panels depending on the time picker choice, get time outs. Is there a control for this?

0 Karma

pj
Contributor

Not using pooled search heads - pooling has proven to be challenging. So we are just using regular SH and indexers. Correct use of summary indexing negates the need to even use hiddensavedsearch in this manner.

0 Karma

richnavis
Contributor

I had lots of problem with this error when I used pooled searchheads.. once I went away from them.. they went away.. What is your architecture?

0 Karma

pj
Contributor

Blast from the past - thanks for the comment reminding me about this. So I actually gave up on this as I could not find a workable solution. After all sorts, of other trial and error, we ended up using summary indexing and then running the dashboards from the summary index. The approach i originally used should have worked, but it doesnt (maybe a bug, who knows). If you keep the fields in your summary down (v. important for speed), you should be able to get a pretty fast render in the dashboard - usually 10secs or so.

0 Karma

cpeteman
Contributor

Did anyone ever fix this problem?

0 Karma

vmorel
Engager

Same issue here, same version and also with a HiddenSavedSearch. No solution so far. It only happens when I change the time to more than 4 hours.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...