Deployment Architecture

distributed search: Application does not exist: search

krugger
Communicator

Hi,

There are 2 splunk servers( A and B) that have differente data and indexes. I have setup distributed search from A to B and B to A.

searches done from A to B: everything is working as expected. I search for an index that only exists in B and I get results.

searches done from B to A: searches for indexes in A fail, and searches for local indexes give out warnings about searches in B failing.

Looking through the search.log I see messages like
ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: Application does not exist: search

However the application exists in both splunk server. And why is every single search being replicated between both splunk servers. If I am looking for something in an index that is in B while I am connected to B is shouldn't replicate the search to A.

I think I should be using clustering instead of distributed search as I plan on having different indexes and data on each of the splunk servers.

Tags (1)
0 Karma
1 Solution

kristian_kolb
Ultra Champion

One of the points of a distributed setup is the performance benefit of spreading the workload of searching over several indexers (indexers do a major part of work in searches).

Also, this will let you use load balancing from the forwarders, which will give a higher resilience, since you can take one indexer down for maintenance, and events will still be forwarded to the other indexer.

If you really want to have separate indexes on separate servers, and to limit where the searches are sent, I guess you could use the splunk_server=serverB search restriction, since Splunk does not inherently know on which indexer a certain index exists.

Or... hmm.. perhaps that field (splunk_server) just filters incoming results from all indexers... in that case have a look at localop

http://docs.splunk.com/Documentation/Splunk/5.0.2/SearchReference/Localop

/K

View solution in original post

0 Karma

kristian_kolb
Ultra Champion

One of the points of a distributed setup is the performance benefit of spreading the workload of searching over several indexers (indexers do a major part of work in searches).

Also, this will let you use load balancing from the forwarders, which will give a higher resilience, since you can take one indexer down for maintenance, and events will still be forwarded to the other indexer.

If you really want to have separate indexes on separate servers, and to limit where the searches are sent, I guess you could use the splunk_server=serverB search restriction, since Splunk does not inherently know on which indexer a certain index exists.

Or... hmm.. perhaps that field (splunk_server) just filters incoming results from all indexers... in that case have a look at localop

http://docs.splunk.com/Documentation/Splunk/5.0.2/SearchReference/Localop

/K

0 Karma

krugger
Communicator

I would have to edit all the searches on both server. They are like 300 of them, I will do if there is no other way

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...