Archive

Why dispatch.fetch is very Slow while retrieving data from Cassandra ?

basilarockiaedw
Path Finder

I am connecting my hunk application(6.4) to datastax cassandra 3.1 to get the results for monitoring and the results took consistently 5 seconds to render though the table has data in hundreds.
I have verified my CassandraERP Connector class also which is hardly taking time in mili seconds to return the response.Could anyone help me in getting this clarified.

Execution costs
Duration (seconds) Component Invocations Input count Output count
0.00 command.fields 4 1 1
0.00 command.search 4 1 1
0.00 command.search.filter 4 - -
2.02 command.stdin 3 - 1
2.00 command.stdin.cpd2sr 2 1 1
0.00 command.stdin.calcfields 1 1 1
2.00 command.stdin.cpd2sr.blocked 1 - -
0.00 command.stdin.kv 1 1 1
0.00 command.stdin.tags 1 1 1
0.00 command.stdin.typer 1 1 1
0.00 command.stdin.fieldalias 1 1 1
0.00 command.stdin.lookups 1 1 1
0.00 dispatch.check_disk_usage 1 - -
0.06 dispatch.createdSearchResultInfrastructure 1 - -
0.04 dispatch.evaluate 1 - -
0.04 dispatch.evaluate.search 1 - -
4.08 dispatch.fetch 6 - -
0.00 dispatch.localSearch 1 - -
0.00 dispatch.preview 1 - -
0.00 dispatch.readEventsInResults 1 - -
0.00 dispatch.stream.local 1 - -
0.00 dispatch.timeline 6 - -
0.03 dispatch.writeStatus 8 - -
0.01 startup.configuration 1 - -
0.03 startup.handoff 1 - -

0 Karma
1 Solution

rdagan_splunk
Splunk Employee
Splunk Employee

Based on the docs http://docs.splunk.com/Documentation/Splunk/latest/Search/ViewsearchjobpropertieswiththeJobInspector
dispatch.fetch = The time spent waiting for or fetching events from search peers.

Since you are not getting the data from the Splunk Indexers, I translate this duration as the total time that includes
Send CQL to Cassandra + Translate the Cassandra results to JSON + Network and Stream all the results back to the Search Head

I am running a similar search on few thousands events and getting the same behavior. The Job inspector shows this message
" This search has completed and has returned 1,000 results by scanning 76,790 events in 6.945 seconds (4.64 dispatch.fetch) "

View solution in original post

kschon_splunk
Splunk Employee
Splunk Employee

I believe that number also includes the time to start a new JVM (not that this should take 4 seconds).

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

Based on the docs http://docs.splunk.com/Documentation/Splunk/latest/Search/ViewsearchjobpropertieswiththeJobInspector
dispatch.fetch = The time spent waiting for or fetching events from search peers.

Since you are not getting the data from the Splunk Indexers, I translate this duration as the total time that includes
Send CQL to Cassandra + Translate the Cassandra results to JSON + Network and Stream all the results back to the Search Head

I am running a similar search on few thousands events and getting the same behavior. The Job inspector shows this message
" This search has completed and has returned 1,000 results by scanning 76,790 events in 6.945 seconds (4.64 dispatch.fetch) "

View solution in original post

basilarockiaedw
Path Finder

Thanks for the response!..In My case though the search results count is very less(1) it is taking 4 to 5 seconds to return the response. In any way can we reduce this time(disatch.fetch) interval?

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

I saw the same behavior on the Datastax VM. Overall - 1 results or 100,000 results gave me the same range between 3 to 5 seconds for the dispatch.fetch

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!