topic Re: Why dispatch.fetch is very Slow while retrieving data from Cassandra ? in Splunk Search

Why dispatch.fetch is very Slow while retrieving data from Cassandra ?

basilarockiaedw — Tue, 29 Sep 2020 11:52:32 GMT

I am connecting my hunk application(6.4) to datastax cassandra 3.1 to get the results for monitoring and the results took consistently 5 seconds to render though the table has data in hundreds.
I have verified my CassandraERP Connector class also which is hardly taking time in mili seconds to return the response.Could anyone help me in getting this clarified.

Execution costs
Duration (seconds) Component Invocations Input count Output count
0.00 command.fields 4 1 1
0.00 command.search 4 1 1
0.00 command.search.filter 4 - -
2.02 command.stdin 3 - 1
2.00 command.stdin.cpd2sr 2 1 1
0.00 command.stdin.calcfields 1 1 1
2.00 command.stdin.cpd2sr.blocked 1 - -
0.00 command.stdin.kv 1 1 1
0.00 command.stdin.tags 1 1 1
0.00 command.stdin.typer 1 1 1
0.00 command.stdin.fieldalias 1 1 1
0.00 command.stdin.lookups 1 1 1
0.00 dispatch.check_disk_usage 1 - -
0.06 dispatch.createdSearchResultInfrastructure 1 - -
0.04 dispatch.evaluate 1 - -
0.04 dispatch.evaluate.search 1 - -
4.08 dispatch.fetch 6 - -
0.00 dispatch.localSearch 1 - -
0.00 dispatch.preview 1 - -
0.00 dispatch.readEventsInResults 1 - -
0.00 dispatch.stream.local 1 - -
0.00 dispatch.timeline 6 - -
0.03 dispatch.writeStatus 8 - -
0.01 startup.configuration 1 - -
0.03 startup.handoff 1 - -

Re: Why dispatch.fetch is very Slow while retrieving data from Cassandra ?

rdagan_splunk — Mon, 28 Nov 2016 17:46:28 GMT

Based on the docs http://docs.splunk.com/Documentation/Splunk/latest/Search/ViewsearchjobpropertieswiththeJobInspector
dispatch.fetch = The time spent waiting for or fetching events from search peers.

Since you are not getting the data from the Splunk Indexers, I translate this duration as the total time that includes
Send CQL to Cassandra + Translate the Cassandra results to JSON + Network and Stream all the results back to the Search Head

I am running a similar search on few thousands events and getting the same behavior. The Job inspector shows this message
" This search has completed and has returned 1,000 results by scanning 76,790 events in 6.945 seconds (4.64 dispatch.fetch) "

Re: Why dispatch.fetch is very Slow while retrieving data from Cassandra ?

kschon_splunk — Mon, 28 Nov 2016 18:21:59 GMT

I believe that number also includes the time to start a new JVM (not that this should take 4 seconds).

Re: Why dispatch.fetch is very Slow while retrieving data from Cassandra ?

basilarockiaedw — Thu, 01 Dec 2016 09:54:35 GMT

Thanks for the response!..In My case though the search results count is very less(1) it is taking 4 to 5 seconds to return the response. In any way can we reduce this time(disatch.fetch) interval?

Re: Why dispatch.fetch is very Slow while retrieving data from Cassandra ?

rdagan_splunk — Thu, 01 Dec 2016 17:45:35 GMT

I saw the same behavior on the Datastax VM. Overall - 1 results or 100,000 results gave me the same range between 3 to 5 seconds for the dispatch.fetch