Splunk Search

Determine which site's search is taking time

Na_Kang_Lim
Path Finder

I have a multisite setup. Each site has 3-4 indexers, with a Replication Factor = 2.

Search Factor is = 1.

When querying for data from all the sites, how to determine which site is taking the longest to search for data?

Labels (1)
0 Karma
1 Solution

livehybrid
SplunkTrust
SplunkTrust

Hi @Na_Kang_Lim 

You may be able to work this out based on the search log:

  • Run your search
  • Click on "Job" > "Inspect Job"
    • Look under "Execution costs" for the "dispatch.stream.remote" section which lists each indexer queried and how long it took.
    • and/or click the "Job Details Dashboard" link on the top of the job inspection page and scroll to the bottom to see "Time Spent Running Search Per Indexer"

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

View solution in original post

gcusello
SplunkTrust
SplunkTrust

hi @Na_Kang_Lim ,

only one additional information: search_factor=1 isn't a good configuration because having one indexer down, you don't have all the data available for searches, at least use SF=2, it's better, even if in this way you must use more storage space.

Ciao.

Giuseppe

0 Karma

isoutamo
SplunkTrust
SplunkTrust
And when you have multisite cluster you are quite probably talking about site_search_factor not about search factor which are still there but valid only inside each sites. https://docs.splunk.com/Documentation/Splunk/9.4.1/Indexer/Multisitearchitecture
SSF and SRF should be at least 2 (usually those are 2 or max 3, depending on how many sites you have).
Then there is also search affinity parameter which told are SHs using all sites or only their own site’s indexers when they are searching.
0 Karma

livehybrid
SplunkTrust
SplunkTrust

Hi @Na_Kang_Lim 

You may be able to work this out based on the search log:

  • Run your search
  • Click on "Job" > "Inspect Job"
    • Look under "Execution costs" for the "dispatch.stream.remote" section which lists each indexer queried and how long it took.
    • and/or click the "Job Details Dashboard" link on the top of the job inspection page and scroll to the bottom to see "Time Spent Running Search Per Indexer"

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

isoutamo
SplunkTrust
SplunkTrust

Here is one blog post about job inspector. https://www.consist.de/de/unternehmen/blog/artikel/Splunk-Suchen-verstehen-mit-dem-Job-Inspector/ If needed google translator do quite good job to translate it if needed. At the end of it are some links which gives couple of excellent other presentations about it.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

May 2026 Splunk Expert Sessions: Security & Observability

Level Up Your Operations: May 2026 Splunk Expert Sessions Whether you are refining your security posture or ...

Network to App: Observability Unlocked [May & June Series]

In today’s digital landscape, your environment is no longer confined to the data center. It spans complex ...

SPL2 Deep Dives, AppDynamics Integrations, SAML Made Simple and Much More on Splunk ...

Splunk Lantern is Splunk’s customer success center that provides practical guidance from Splunk experts on key ...