Why dispatch.fetch is very Slow while retrieving data from Cassandra ?

[basilarockiaedw]

I am connecting my hunk application(6.4) to datastax cassandra 3.1 to get the results for monitoring and the results took consistently 5 seconds to render though the table has data in hundreds.
I have verified my CassandraERP Connector class also which is hardly taking time in mili seconds to return the response.Could anyone help me in getting this clarified.

Execution costs
Duration (seconds) Component Invocations Input count Output count
0.00 command.fields 4 1 1
0.00 command.search 4 1 1
0.00 command.search.filter 4 - -
2.02 command.stdin 3 - 1
2.00 command.stdin.cpd2sr 2 1 1
0.00 command.stdin.calcfields 1 1 1
2.00 command.stdin.cpd2sr.blocked 1 - -
0.00 command.stdin.kv 1 1 1
0.00 command.stdin.tags 1 1 1
0.00 command.stdin.typer 1 1 1
0.00 command.stdin.fieldalias 1 1 1
0.00 command.stdin.lookups 1 1 1
0.00 dispatch.check_disk_usage 1 - -
0.06 dispatch.createdSearchResultInfrastructure 1 - -
0.04 dispatch.evaluate 1 - -
0.04 dispatch.evaluate.search 1 - -
4.08 dispatch.fetch 6 - -
0.00 dispatch.localSearch 1 - -
0.00 dispatch.preview 1 - -
0.00 dispatch.readEventsInResults 1 - -
0.00 dispatch.stream.local 1 - -
0.00 dispatch.timeline 6 - -
0.03 dispatch.writeStatus 8 - -
0.01 startup.configuration 1 - -
0.03 startup.handoff 1 - -

[kschon_splunk]

I believe that number also includes the time to start a new JVM (not that this should take 4 seconds).

[rdagan_splunk · Accepted solution]

Based on the docs http://docs.splunk.com/Documentation/Splunk/latest/Search/ViewsearchjobpropertieswiththeJobInspector
dispatch.fetch = The time spent waiting for or fetching events from search peers.

Since you are not getting the data from the Splunk Indexers, I translate this duration as the total time that includes
Send CQL to Cassandra + Translate the Cassandra results to JSON + Network and Stream all the results back to the Search Head

I am running a similar search on few thousands events and getting the same behavior. The Job inspector shows this message
" This search has completed and has returned 1,000 results by scanning 76,790 events in 6.945 seconds (4.64 dispatch.fetch) "

[basilarockiaedw]

Thanks for the response!..In My case though the search results count is very less(1) it is taking 4 to 5 seconds to return the response. In any way can we reduce this time(disatch.fetch) interval?

[rdagan_splunk]

I saw the same behavior on the Datastax VM. Overall - 1 results or 100,000 results gave me the same range between 3 to 5 seconds for the dispatch.fetch