Getting Data In

Distributed Search: After making a search head also an indexer, why are indexes created on the search head instead found on another indexer?

howyagoin
Contributor

Stupid question time.

I've got a pretty simple setup. Search head, two indexers. Everything works great.

Except that my search head is overly resourced for being a search head, and I'd like to add some indexing to it.

If I go into the Settings and create an Index, I see the directory appear on the Search Head just fine, usual location, but, as soon as I start actually indexing data, by, say, indexing a file or directory, the data appears on one of my Indexers and NOT on the Search Head.

Alpha is the indexer, Beta is an indexer and Gamma is an indexer.

I create the index on Alpha, see it on the file system. I then read the file/data in and assign it to the index, by name, that only exists on Alpha.

However, at the same time, the index is created on Beta, and when the data is read in, it is actually indexed on Beta.

A Splunk search for the data shows the splunk_server as being "beta"...

I have no idea how I wound up setting that up, nor why it's not going to Gamma as well. No clusters have been set up, no replication.

I just want to put Alpha to a bit better use.

What obvious thing am I missing?!

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

I'm guessing your search head has been configured to forward to your indexers when setting up the system to store _internal logs where they belong.

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

I'm guessing your search head has been configured to forward to your indexers when setting up the system to store _internal logs where they belong.

howyagoin
Contributor

Well, it hasn't been, at least intentionally. I configured all parts of these and at no time did I specifically tell Alpha to send all data to Beta. Indeed, there is no reference to "beta" anywhere in the configuration files for Alpha other than in the distsearch.conf.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Just to be sure, what do you see when you go to Settings -> Forwarding and receiving -> Configure forwarding on your search head?

howyagoin
Contributor

BINGO. Give the man a gold star, a coffee, a single malt. It apparently has been configured for the IP address of beta, and not beta as a hostname, which is why no grepping would show it. ARGH. This explains things. I'll delete this and see if things go back to what I wanted. Thank you...turns out this was "outputs.conf" and I can't believe I didn't check there.

jimodonald
Contributor

You really can't use the Web UI to create indexes in a distributed index cluster. You need to modify the indexes.conf on each indexer and then restart them.

http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Setupmultipleindexes#Create_and_edit_inde...

0 Karma

howyagoin
Contributor

Interesting, if counter-intuitive. I am certain that in previous versions of Splunk this was not always the case. We used to have a combined indexer/search head, which also had distributed search enabled, and indexes appeared on the local system just fine. This is something "new" which appears to have come up with a recent update.

I'll try this - will have to restart the Search Head to test...will report back. Thanks.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Is this even an indexer cluster or just traditional distributed search?

0 Karma

howyagoin
Contributor

So, I don't mind the change of the post title, ppablo_splunk, but it's not accurate. I don't know that I've made the Search Head an indexer - I WANT to do that, and I want to do it correctly. Your edit isn't accurate, but thanks for trying to tidy up the wording otherwise.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...