Searching local indexes on a search head?

[mfrost8]

We have a test server that's indexing data locally (with sufficient license to do so). For some development effort, we also have a need to make it a search head against our 4 production search peers.

Yes, the index where all the test data is going on the test server is "main".

I modified the configuration on this test server to use the search peers which has worked fine, but now searches on this test server no longer see the local events at all -- only events from the search peers.

I'm not able to find much of any information about local indexes on a search head, so I'm not clear if this is because you can't really search local indexes once an instance becomes a search head, or if it's because the index on the search head is "main" and that index is also on the search peers and Splunk won't work with both.

I may have no option but to disable the search peers on the test server, but I'd like to understand what the problem is.

=== UPDATE
OK, so apparently the characterization of this issue that I was given by the user wasn't accurate. They aren't actually using 'main', but 2 separate unique indexes locally. And also, apparently, their searches are working, just taking a bit longer. I pointed out that it's because now with more open-ended searches, Splunk has more indexes/peers to look at and potentially more data to return so refining the locations that are searched is in order.

This would actually have been more interesting had their been events in 'main', but I suspect that Splunk might have done the right thing there too.

Thanks for pointing out the splunk_server field!

Thanks!

[bmunson_splunk]

This seems to be a recent change that came in with v6.4. I have found if you add the following to the beginning of your search it will include the search head and all other servers.

 splunk_server=*

[mfrost8]

Huh. Interesting. Nice to know it can handle it, but also that it knows it's kind of not right. Thanks.

[MuS]

Just as addition: you can add localhost:8089 as search peer. But it will through some errors in splunkd.log:

splunkd.log:09-12-2014 07:19:36.480 +0200 WARN  DistributedPeerManager - Unable to distribute to peer named somehostname.FQDN.here at uri https://localhost:8089 because peer has status = "Duplicate Servername".

[mfrost8]

I do indeed get the splunk_server values I would expect, but as I just mentioned/updated, it looks like the events in question are not in 'main'. I suspect that Splunk would probably "do the right thing" in that situation and return events from all 'main' indexes.

I wouldn't think that Splunk could be explicitly named as its own search peer. It kind of is already, isn't it?

[MuS]

What just came up my mind: can a search head be its own search peer? In other words, what will happen if you add localhost:8089 as search peer on this search head?

[tom_frotscher · Accepted solution]

If you start this search on your test server (over a long time duration):

index=main

What do you get in the field splunk_server? Are there only your 4 production machines in there, or also the search head?

[mfrost8]

Per the answer, when I ran the index=main search over a long time, I did only get the 4 production search peers in the splunk_server field. Per my update, since I found out later that the test server had only unique indexes on it and never used 'main', that actually made sense. Seems to be doing what I need/expect. Thanks!

[yannK]

Responded by tom_frotscher as a comment. Please accept the answer to mark the question as resolved.

you can specify the splunk indexer name "splunk_server"
index=* splunk_server=<mysearchheadhostname>

[bandit]

Seems this behavior changed at some point with one of the Splunk releases. It used to default to search locally even if distributed search peers were defined. i.e. it would search local and remote indexes by default. It's a pain to have to prefix searches with "splunk_server=" for environments with a test or staging search head/indexer combo that also searches production indexers. I'd like to see the as an option in a .conf file to turn local search on/off.

[bandit]

Splunk, please add this option back like it was in previous releases!