Deployment Architecture

distributed search: Application does not exist: search

krugger
Communicator

Hi,

There are 2 splunk servers( A and B) that have differente data and indexes. I have setup distributed search from A to B and B to A.

searches done from A to B: everything is working as expected. I search for an index that only exists in B and I get results.

searches done from B to A: searches for indexes in A fail, and searches for local indexes give out warnings about searches in B failing.

Looking through the search.log I see messages like
ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: Application does not exist: search

However the application exists in both splunk server. And why is every single search being replicated between both splunk servers. If I am looking for something in an index that is in B while I am connected to B is shouldn't replicate the search to A.

I think I should be using clustering instead of distributed search as I plan on having different indexes and data on each of the splunk servers.

Tags (1)
0 Karma
1 Solution

kristian_kolb
Ultra Champion

One of the points of a distributed setup is the performance benefit of spreading the workload of searching over several indexers (indexers do a major part of work in searches).

Also, this will let you use load balancing from the forwarders, which will give a higher resilience, since you can take one indexer down for maintenance, and events will still be forwarded to the other indexer.

If you really want to have separate indexes on separate servers, and to limit where the searches are sent, I guess you could use the splunk_server=serverB search restriction, since Splunk does not inherently know on which indexer a certain index exists.

Or... hmm.. perhaps that field (splunk_server) just filters incoming results from all indexers... in that case have a look at localop

http://docs.splunk.com/Documentation/Splunk/5.0.2/SearchReference/Localop

/K

View solution in original post

0 Karma

kristian_kolb
Ultra Champion

One of the points of a distributed setup is the performance benefit of spreading the workload of searching over several indexers (indexers do a major part of work in searches).

Also, this will let you use load balancing from the forwarders, which will give a higher resilience, since you can take one indexer down for maintenance, and events will still be forwarded to the other indexer.

If you really want to have separate indexes on separate servers, and to limit where the searches are sent, I guess you could use the splunk_server=serverB search restriction, since Splunk does not inherently know on which indexer a certain index exists.

Or... hmm.. perhaps that field (splunk_server) just filters incoming results from all indexers... in that case have a look at localop

http://docs.splunk.com/Documentation/Splunk/5.0.2/SearchReference/Localop

/K

0 Karma

krugger
Communicator

I would have to edit all the searches on both server. They are like 300 of them, I will do if there is no other way

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...