Improve TSTATS performance (dispatch.localSearch) command with more Indexers(Search nodes)?

[robertlynch2020]

HI

I have the following tstat command that takes ~30 seconds (dispatch.localSearch) is the main slowness .

I have been told to add more indexers to help with this, as the accelerated Datamodel is held on the search head(I think) and more indexers will help to split the dispatch.localSearch - This is the theory??

What i dont know is do i need to set up a cluster index or a non cluster index do i need with full replication for this to help?

So at the moment, i have one Splunk install on one machine.

I can get more machines if needed. I need some advice on what is the best way forward. - Thanks in advance.

P,S I cant really change the search, what i would like to know is that will more indexers(search nodes)

| tstats summariesonly=true chunk_size=1000000 max(MXTIMING.Elapsed) AS Elapsed max(MXTIMING.CPU) AS CPU max(MXTIMING.CPU_PER) AS CPU_PER max(MXTIMING.Memory_V2) AS Memory  FROM datamodel=MXTIMING_V9 WHERE 
    host=QCST_RSAT_V41 
    AND MXTIMING.Elapsed > 1 OR 1=1
    GROUPBY _time MXTIMING.Machine_Name MXTIMING.Context+Command MXTIMING.NPID MXTIMING.MXTIMING_TYPE_DM source MXTIMING.UserName2 span=1s

The inspect is this

Execution costs
Duration (seconds)      Component   Invocations Input count Output count
    0.14     .execute_output.flush_stats    1   -   -
    0.49     .execute_output.merge_stats    1   -   -
    37.05    command.tstats 328 62,772  62,772
    34.70    command.tstats.query_tsidx 120 -   -
    1.29     command.tstats.execute_input   164 62,772  -
    0.99     command.tstats.execute_output  1   -   -
    0.00     dispatch.check_disk_usage  4   -   -
    0.00     dispatch.createdSearchResultInfrastructure 1   -   -
    0.00     dispatch.evaluate  1   -   -
    0.00     dispatch.evaluate.tstats   1   -   -
    0.00     dispatch.evaluate.noop 1   -   -
    27.40    dispatch.fetch 164 -   -
    34.64    dispatch.localSearch   1   -   -
    0.00     dispatch.optimize.FinalEval    1   -   -
    0.02     dispatch.optimize.matchReportAcceleration  1   -   -
    0.04     dispatch.optimize.optimization 1   -   -
    0.00     dispatch.optimize.reparse  1   -   -
    0.00     dispatch.optimize.toJson   1   -   -
    0.00     dispatch.optimize.toSpl    1   -   -
    6.09     dispatch.preview   16  -   -
    5.24     dispatch.preview.tstats.execute_output 16  -   -
    0.70     dispatch.preview.write_results_to_disk 16  -   -
    34.77    dispatch.stream.local  163 -   -
    0.08     dispatch.writeStatus   41  -   -
    0.15     startup.configuration  1   -   -
    0.18     startup.handoff    1   -   -

[martin_mueller · Accepted solution]

Assuming you really can't serve your use case with a more efficient search - no way of knowing without knowing the use case -, spreading the search load over multiple indexers would indeed make the search run faster. Datamodel acceleration is performed by, stored on, and searched by the indexers; having many of them will divide the load between them and speed things up.

Failing that, getting faster IO for your single machine may help too.

[martin_mueller]

Yup. When you run a search over recent data, ie during which forwarders were loadbalancing between the two indexers, do open the job inspector and look at the dispatch.remote area of the histogram near the bottom. You should see both indexers reporting, ideally each spending half the time and returning half the data.

[robertlynch2020]

Hi

Thanks for this, i have gotten this working for index=mlc_live and i can see dispatch.remote working on the search head 🙂

But when i try to access the datamodels on the searchhead i am getting cant find datamodel!

So just for the record, i created them on the search head and then un-accelerated them - this seemed to work.

    0.04     dispatch.stream.remote.dell425srv_5000 25  -   149,257
    0.04     dispatch.stream.remote.hp4000_5000 39  -   227,330

---

    0.13     command.tstats 149 99  99
    0.04     command.tstats.execute_input   76  99  -
    0.03     command.tstats.query_tsidx 16  -   -
    0.00     command.tstats.execute_output  1   -   -
    0.00     dispatch.check_disk_usage  1   -   -
    0.01     dispatch.createdSearchResultInfrastructure 1   -   -
    0.01     dispatch.evaluate  3   -   -
    0.00     dispatch.evaluate.noop 3   -   -
    0.00     dispatch.evaluate.tstats   3   -   -
    0.16     dispatch.fetch 76  -   -
    0.00     dispatch.localSearch   1   -   -
    0.01     dispatch.optimize.FinalEval    3   -   -
    0.07     dispatch.optimize.matchReportAcceleration  3   -   -
    0.02     dispatch.optimize.optimization 3   -   -
    0.00     dispatch.optimize.reparse  3   -   -
    0.01     dispatch.optimize.toJson   3   -   -
    0.00     dispatch.optimize.toSpl    3   -   -
    0.07     dispatch.parserThread  66  -   -
    0.01     dispatch.stream.local  8   -   -
    0.09     dispatch.stream.remote 64  -   376,587
    0.04     dispatch.stream.remote.dell425srv_5000 25  -   149,257
    0.04     dispatch.stream.remote.hp4000_5000 39  -   227,330
    0.06     dispatch.writeStatus   14  -   -
    0.10     startup.configuration  7   -   -
    0.81     startup.handoff    7   -   -

[martin_mueller]

Without a cluster you have two options:

1. Connect forwarders to all indexers, wait for data to naturally balance
2. Take down all indexers, manually move buckets from old indexer to new indexers to balance data, possibly do some manual steps such as increasing the next bucket id on the new indexers (not sure if that's still necessary), bring back up all indexers, connect forwarders to all indexers, enjoy balanced data right away

1 is lazy but takes time.
2 is a manual process, but doesn't require waiting for weeks or months.

[robertlynch2020]

Thanks again (I think this is the last question on this 🙂

For options 1

Do you mean the the forwarder will send parts of the data to each indexer or each forwarder will send all data to each indexer?

As i initially understood that sections of the data will go to each indexer and then on search time it will be retried.
Below i can see my forwarder switching between each indexer in the log. in this case i have 2 indexers so they are getting 50% each. Then i was hoping in search time they would return 50% each to the search head , thus getting the speed up i am looking for.

[tcpout:my_LB_indexers]
server=10.25.5.169:5997,10.25.53.57:5997
maxQueueSize=500MB
autoLBVolume=1048

[inputproc]
max_fd = 10000

11-09-2018 19:53:50.861 +0100 INFO  TcpOutputProc - Connected to idx=10.25.5.169:5997
11-09-2018 19:54:20.838 +0100 INFO  TcpOutputProc - Connected to idx=10.25.53.57:5997
11-09-2018 19:55:21.300 +0100 INFO  TcpOutputProc - Connected to idx=10.25.5.169:5997
11-09-2018 19:55:51.271 +0100 INFO  TcpOutputProc - Connected to idx=10.25.53.57:5997
11-09-2018 19:56:20.968 +0100 INFO  TcpOutputProc - Connected to idx=10.25.5.169:5997
11-09-2018 19:57:51.031 +0100 INFO  TcpOutputProc - Connected to idx=10.25.53.57:5997
11-09-2018 19:58:50.753 +0100 INFO  TcpOutputProc - Connected to idx=10.25.5.169:5997
11-09-2018 19:59:21.301 +0100 INFO  TcpOutputProc - Connected to idx=10.25.53.57:5997

[martin_mueller]

You'd let your forwarders loadbalance between the three indexers, and eventually you will have a third of your data in each indexer - tripling your search speed in ideal cases.
Without any data in the two new indexers there's nothing for them to do, and no reduction of work for the original instance - same search speed.

Indexer cluster replication isn't necessary for this, as previously stated. However, if you need immediate changes without waiting for natural balancing of data to occur, you can of course create an indexer cluster and attempt a data rebalance: http://docs.splunk.com/Documentation/Splunk/7.2.0/Indexer/Rebalancethecluster
This may take a while, depending on how much data you have and how wide your network link between the instances is.
Thinking forward though, you'll still need to let all forwarders send data to all indexers - otherwise you will lose that newly gained balance eventually. Additionally, once you have a many-indexers-cluster you'll want to separate your existing instance into a standalone search head without indexing duties. In short, make this image happen 😃 http://docs.splunk.com/Documentation/Splunk/7.2.0/Deploy/Indexercluster#Indexer_cluster_with_individ...

[robertlynch2020]

Hi

Thanks again for the great answer 🙂

So can i do what i want without a cluster? [I have set up my forwarder to send data to 3 indexers, 2 new 1 original]
I am looking at this "how to deploy a non-clustered distributed search topology"
http://docs.splunk.com/Documentation/Splunk/7.2.0/DistSearch/Overviewofconfiguration

In my case i have an app with an index (MLC_LIVE) and a Datamodel (MXTIMING).
I am assuming in order for my new indexers to do anything they have to have to copy the app files? so each indexer knows what MLC_LIVE is and it then has to create MXTIMING datamodel?

When i run my search do i have to tell my search head i have other indexers or it i set it up right this will be "under the hood" for me?

Thanks again
Rob

[robertlynch2020]

Thanks for the great answer :).

I will try and report back

[martin_mueller]

Do the new indexers have buckets they are primaries for that participate in your search?

[robertlynch2020]

Hi

In fact I think i am missing a step.

I had one search head, one indexer on one machine (Inside the same install) = Main Install.

I have exploded 2 more full Splunk installs on 2 new machines, I have added them them in the main install via Distributed search - > Search peers. [Please note the forwarders are only forwarding to the main install - I am not sure if I have to change this?]

So, I am unsure what to do, I was hoping the main install would see the 2 new indexers as extra resource and be able to use them, however i think my understanding is not good enough!!!

When you say "Do the new indexers have buckets" - I am what you mean.

This is why i was asking about Replication
http://docs.splunk.com/Documentation/Splunk/7.2.0/Indexer/Bucketsandclusters

Cheers
Robert

[martin_mueller]

Replication in an indexer cluster is unrelated to search performance.

[robertlynch2020]

Thanks for you help.

So i have set up 2 new indexers on 2 different machine and added them in the Distributed search - > Search peers. Replication status, Health status, Health check failures all good 🙂 . Then a restart.

But when i run a search any search, the CPU on the new boxes stay at 0.0% and the time is the same, so is there something i have to do?

Cheers in advance

[martin_mueller]

Your system usage values suggest you can run more searches in parallel, but single searches can still be pegged.

Having multiple indexers allows you to run single searches in parallel = faster.
There are settings to increase search parallelization on a single instance, but I'm not sure if tstats from datamodels will benefit or not - check the docs for that.

[robertlynch2020]

Thanks for the answer - I will try this out

[robertlynch2020]

I have multiple indexers, but do i have to make them a cluster with 100% replication as well for a search to work like this?

[robertlynch2020]

Hi

Thanks for your replay 🙂

In fact i have gotten system team to monitor the machine and it looks like the machine is not working hard at all. IO = 14% - Disk Access = 10% and CPU = 30%.

I posed a question on it
https://answers.splunk.com/answers/696751/tstat-search-taking-a-long-time-io-cpu-and-disk-ac.html

So how can i tell that the one Indexer i have if FULL and i need more indexers, before i go and make more etc..?
Is that possible

Cheers
Rob