Getting Data In

SimpleResultsTable: Splunkd times out

jagresz
Explorer

Hey,

I want to run a query which has many results (at about 100k) and want run postprocesses to create stats from the main search's result:

search index=something | table field1,field2,etc

postprocess1= stats sum(field1) by action

postprocess2= stats sum(field2) by host

etc...

I can see in Jobs menu the main search completes and gets "done" status after couple of minutes, but after 4 minutes I get "[SimpleResultsTable module] Splunkd daemon is not responding: ('The read operation timed out',)" error message on GUI."

Is there a limitation within Splunk config about postprocesses or about number of results can be display?

(Splunk version 5.0.2 with Sideviewutils 1.3.5)

Thanks!

Cheers,
Istvan

0 Karma

jagresz
Explorer

So thats why think to myself to use a search querying all events and postprocesses with subsearches to filter out events for each subchapter.
Maybe I was wrong and there's a more useful case for that. What do you think?
Thanks!
Istvan

0 Karma

jagresz
Explorer

OK, let me explain what I'd like do in my report from an other aspect. This report has several chapter each begins with a summary followed by several sub-chapters based on summarized events but with different content like:
Chapter 1 - all logons count by successful and failed events
SubChapter 1.1 - all failed logon events by username
SubChapter 1.2 - all successful logons by hosts
etc.
What is in mind that once this summary has been generated we have all the data for sub-chapters needed. But each of them have to filter out non-relevant data and create a stat for themselves. (be cont.)

0 Karma

aholzer
Motivator

jagresz, using postprocess does not equate to doing a subsearch. When you do a join as part of a search, that is considered a subsearch. Because you need to have a separate search run and then perform a join, this second search is known as the subsearch.

postprocess just takes the results from your underlying search and applies new commands to it, so the subsearch limits do not apply.

I can't say I'm familiar with the older versions of sideview utils, but if you are seeing results in the flashtime view, and not in your dashboard, then it's probably something to do with the module you chose.

0 Karma

jagresz
Explorer

aholzer, you're absolutely right, I started to sniff about subsearch limits after I found Answer#64157. My code looks more like this:

search index=something | table field1,field2,etc
postprocess1= | search tag=tag1 | stats sum(field1) by action
postprocess2= | search tag=tag2 | stats sum(field2) by host

The aboves for shorter time ranges run just fine. When I run in flashtimeline view there's no error message I get results I expected (limited to 10000 events as defined in limits.conf)

0 Karma

aholzer
Motivator

With regards to your question:
"Is there a limitation within Splunk config about postprocesses or about number of results can be display?"

There are limitations for subsearches and graphs with regards to how many results you can have / display. But from your example code it doesn't look like you are doing either of these things.

0 Karma

aholzer
Motivator

From your example code it doesn't look like you are using any subsearches, so changing those parameters in your limits.conf wouldn't have an effect.

Why don't you test by limiting the amount of time for your search to something small and see if you get results. For example, if you are running for 1 week, then try running for a day; if you are running for a day try running for an hour. If you get data to display then you'll have a starting point.

When you run your search in the flashtimeline view do you get the same error or do you get all the results that you expect?

0 Karma

jagresz
Explorer

Yes, this is a single head/indexer setup and a monthly report generation causes this timeout. Load is quite high (above 5) and consumes over 90% CPU time during the generation process.

You're right Drainy, I've found a similar problem here on Answers without any answers:
http://splunk-base.splunk.com/answers/64157/what-limit-causes-the-read-operation-timed-out
I've also tried to play around with 'subsearch' stanza in limits.conf but no effects here too.

0 Karma

Drainy
Champion

It doesn't look like a sideview utils issue to me. Is this a single head/indexer setup and is the box under load? Splunkd not responding can often be because the box is under load, I believe there is a hard coded timeout between the Webui and splunkd. Otherwise there have been a few others posting recently about not responding issues

0 Karma

jagresz
Explorer

I can't agree the license agreement since this app is not for internal use only.. is there an other way to fix things? Is this error a limitation of Sideview 1.3?

Thanks!

0 Karma

aholzer
Motivator

You may want to try upgrading your Sideview Utils installation. The current version is 2.4.3.

You can get the latest version from here: http://sideviewapps.com/apps/sideview-utils/download-full-version-internal-use/

Then use the Splunk GUI to install the download by going to Manager > apps > "Install app from file"

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...