What exactly is 'dispatch.fetch' and 'dispatch.stream.local' and how do I lower its processing time?

[sjanwity]

I have a report which searches through an index of extracted database rows and performs transaction on them as well as some other eval processing, such as flattening multivalue fields and establishing a new column based on the result. The database is big and grows day by day with the new changes being fed into the index through a 30-min cron job on splunk.

The search was taking quite long so I moved the transaction command to another process as detailed in http://answers.splunk.com/answers/187124/collecting-results-from-a-transaction-command-does.html but the search is still taking a while:

This search has completed and has returned 10,000 results by scanning 363,488 events in 212.374 seconds.

Execution costs

Duration (seconds)      Component   Invocations Input count Output count
    22.146  command.eval    288 2,907,896   2,907,896
    0.033   command.fields  36  363,487 363,487
    11.998  command.presort 1   363,487 293,488
    94.876  command.search  288 2,544,414   2,907,901
    85.51   command.search.kv   32  -   -
    7.297   command.search.filter   252 -   -
    1.431   command.search.rawdata  32  -   -
    0.036   command.search.fieldalias   32  363,488 363,488
    0.036   command.search.typer    36  363,488 363,488
    0.035   command.search.tags 36  363,488 363,488
    0.032   command.search.calcfields   32  363,488 363,488
    0.032   command.search.lookups  32  363,488 363,488
    0.02    command.search.summary  36  -   -
    0   command.search.index.usec_1_8   2   -   -
    0   command.search.index.usec_4096_32768    8   -   -
    0   command.search.index.usec_512_4096  23  -   -
    0   command.search.index.usec_64_512    5   -   -
    0   command.search.index.usec_8_64  15  -   -
    0.199   command.sort    1   1,550,569   10,000
    15.166  command.table   1   363,487 1,914,056
    0.014   dispatch.check_disk_usage   14  -   -
    0.054   dispatch.createProviderQueue    1   -   -
    13.166  dispatch.emit_prereport_files   32  -   -
    0.323   dispatch.evaluate   1   -   -
    0.329   dispatch.evaluate.search    8   -   -
    0.008   dispatch.evaluate.eval  8   -   -
    0.001   dispatch.evaluate.sort  1   -   -
    0.001   dispatch.evaluate.table 1   -   -
    116.744 dispatch.fetch  37  -   -
    50.53   dispatch.preview    7   -   -
    24.696  dispatch.preview.command.table  7   840,261 4,027,761
    23.473  dispatch.preview.command.presort    7   840,261 675,262
    0.875   dispatch.preview.command.sort   7   3,170,000   70,000
    0.397   dispatch.preview.write_results_to_disk  7   -   -
    0.001   dispatch.reduce 1   -   -
    116.736 dispatch.stream.local   36  -   -
    2.191   dispatch.writeStatus    42  -   -
    0.094   startup.handoff 1   -   -

as you can see the dispatch is taking the lion's share of processing time. How can I optimise this?

[martin_mueller · Accepted solution]

That dispatch.stream.local is time spent on streaming data from your local indexer. In other words, it looks like it's doing about 3000 events per second... if that's a dense search then you may want to get faster cores on that machine. What's your hardware specs?

http://docs.splunk.com/Documentation/Splunk/6.2.0/Capacity/HowsearchtypesaffectSplunkEnterpriseperfo...

You might save some time by turning preview off, see dispatch.preview eating a nice chunk on top. That might not matter for the scheduled executions though.

search.kv is eating a lot as well, consider | fields'ing your data before piping it into the transaction to only extract fields you actually need.

[sjanwity]

that fields tip did the trick, thanks! Do you have any other optimisation pointers? I also do a table and sort on the end result to display data better. Is there a cheaper way of re-arranging the columns and sorting them?

[MuS]

have a look at this nice post http://answers.splunk.com/answers/172275/how-do-optimizations-for-field-based-searches-work.html to lear more about search optimizations

[sjanwity]

Note: in my splunk search, I generally want splunk to return everything. I have a few form-based input filters but by default they are set to '*'. Is there a way to quicken the search with this?