distributed search: Application does not exist: search

[krugger]

Hi,

There are 2 splunk servers( A and B) that have differente data and indexes. I have setup distributed search from A to B and B to A.

searches done from A to B: everything is working as expected. I search for an index that only exists in B and I get results.

searches done from B to A: searches for indexes in A fail, and searches for local indexes give out warnings about searches in B failing.

Looking through the search.log I see messages like
ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: Application does not exist: search

However the application exists in both splunk server. And why is every single search being replicated between both splunk servers. If I am looking for something in an index that is in B while I am connected to B is shouldn't replicate the search to A.

I think I should be using clustering instead of distributed search as I plan on having different indexes and data on each of the splunk servers.

[kristian_kolb · Accepted solution]

One of the points of a distributed setup is the performance benefit of spreading the workload of searching over several indexers (indexers do a major part of work in searches).

Also, this will let you use load balancing from the forwarders, which will give a higher resilience, since you can take one indexer down for maintenance, and events will still be forwarded to the other indexer.

If you really want to have separate indexes on separate servers, and to limit where the searches are sent, I guess you could use the splunk_server=serverB search restriction, since Splunk does not inherently know on which indexer a certain index exists.

Or... hmm.. perhaps that field (splunk_server) just filters incoming results from all indexers... in that case have a look at localop

http://docs.splunk.com/Documentation/Splunk/5.0.2/SearchReference/Localop

/K

[krugger]

I would have to edit all the searches on both server. They are like 300 of them, I will do if there is no other way