topic After adding a new Splunk server in a distributed environment, why does it not show up in results unless I include splunk_server=*? in Splunk Search

After adding a new Splunk server in a distributed environment, why does it not show up in results unless I include splunk_server=*?

louieb3 — Tue, 01 Sep 2015 16:53:24 GMT

I recently added a new splunk server in a distributed environment. Now, when I do this search:

index=os earliest="09/01/2015:09:30:00" latest="09/01/2015:09:35:00" | timechart count by splunk_server

the new splunk server does not show up in the results. However, if I do this search,

index=os splunk_server=* earliest="09/01/2015:09:30:00" latest="09/01/2015:09:35:00" | timechart count by splunk_server

then, it shows up.

Can anyone tell me why? I have the search load-balanced so I have about the same number of events going into each indexer.
Thank you in advance.

Re: After adding a new Splunk server in a distributed environment, why does it not show up in results unless I include splunk_server=*?

woodcock — Tue, 01 Sep 2015 18:21:34 GMT

What is inside distsearch.conf? How did you add the Indexer? Are you using Search Head Pooling?

See this question, too:

http://answers.splunk.com/answers/221468/search-returns-zero-results-searchlog-reports-dist.html

Re: After adding a new Splunk server in a distributed environment, why does it not show up in results unless I include splunk_server=*?

louieb3 — Tue, 01 Sep 2015 19:09:50 GMT

That was it. I looked at distsearch.conf and saw that all of my indexers except for the new one was in the [distributedSearch:dmc_group_indexer] stanza.

I went into the Distributed Management Console, under Remote instances, edited the Server Role for the new indexer (it was already configured as an indexer), saved it, and then applied the changes and voila, issue resolved. Thanks, woodcock!

To answer your question, in distsearch.conf, I had the stanzas [distributedSearch] which contained all of the indexers and [distributedSearch:dmc_group_indexer] which also contained a list of my indexers except for the recently added one.