We have a test server that's indexing data locally (with sufficient license to do so). For some development effort, we also have a need to make it a search head against our 4 production search peers.
Yes, the index where all the test data is going on the test server is "main".
I modified the configuration on this test server to use the search peers which has worked fine, but now searches on this test server no longer see the local events at all -- only events from the search peers.
I'm not able to find much of any information about local indexes on a search head, so I'm not clear if this is because you can't really search local indexes once an instance becomes a search head, or if it's because the index on the search head is "main" and that index is also on the search peers and Splunk won't work with both.
I may have no option but to disable the search peers on the test server, but I'd like to understand what the problem is.
=== UPDATE
OK, so apparently the characterization of this issue that I was given by the user wasn't accurate. They aren't actually using 'main', but 2 separate unique indexes locally. And also, apparently, their searches are working, just taking a bit longer. I pointed out that it's because now with more open-ended searches, Splunk has more indexes/peers to look at and potentially more data to return so refining the locations that are searched is in order.
This would actually have been more interesting had their been events in 'main', but I suspect that Splunk might have done the right thing there too.
Thanks for pointing out the splunk_server field!
Thanks!
... View more