topic Re: Search head and Indexer has different indexes list in Deployment Architecture

Search head and Indexer has different indexes list

moonyoungjung — Thu, 29 Aug 2019 03:20:07 GMT

I have a question about indexes.
In my environment,

search head cluster is 5ea,
indexer peer node 20ea,
indexer cluster master 1ea and
heavy forwarder etc

When I check indexes list in indexer peer node,
every indexer has the same 30 indexes in the list,
but every search peer has different indexes list.
Some "search head " has indexes 10 list of it
the other has 15 list in setting -> indexes

This is a problem when I added data from search head
I want to add data from B indexes but there is no list in indexes

how can I sync list from indexer?

Re: Search head and Indexer has different indexes list

inventsekar — Thu, 29 Aug 2019 05:31:38 GMT

When I check indexes list in indexer peer node, 
every indexer has the same 30 indexes in the list

Only search head got the splunk GUI. the splunk indexer got no splunk GUI.
only indexer will have indexes. the search head handles only the search requests(and few more functionalities)
From Search head, it internally connects to indexer and gets the list of indexer.

i assume yours is a distributed but not clustered environment.
so, it may be possible that, one search head(SH1) connects an indexer(IDX1), whereas another search head(SH2) may connect to another indexer(IDX2). The indexes configured on IDX1 may be different from IDX2.

Re: Search head and Indexer has different indexes list

moonyoungjung — Wed, 30 Sep 2020 01:57:11 GMT

Thank you for your answer.

I was check Distributed search » Search peers list
SH1 and SH2 have the same list in it.

what conf file should i fix?

This is shclustering stanza my server.conf in SH1, SH2, SH3

SH1 etc/system/local/server.conf
[shclustering]
conf_deploy_fetch_url = http://x.x.x.36:8089
disabled = 0
mgmt_uri = http://x.x.x.31:8089
pass4SymmKey = ****************=
replication_factor = 2
id = A154BD4B-2199-44E0-856F-4781DA470875
shcluster_label = shcluster
election = 1
mode = member

SH2 etc/system/local/server.conf
[shclustering]
conf_deploy_fetch_url = http://x.x.x.36:8089
disabled = 0
mgmt_uri = http://x.x.x.32:8089
pass4SymmKey = *******************=
replication_factor = 2
id = A154BD4B-2199-44E0-856F-4781DA470875
shcluster_label = shcluster
election = 1
mode = member

SH3 etc/system/local/server.conf
[shclustering]
conf_deploy_fetch_url = http://x.x.x.36:8089
disabled = 0
mgmt_uri = http://x.x.x.33:8089
pass4SymmKey = ***********************=
replication_factor = 2
id = A154BD4B-2199-44E0-856F-4781DA470875
shcluster_label = shcluster
election = 1
mode = member

Re: Search head and Indexer has different indexes list

moonyoungjung — Thu, 29 Aug 2019 06:27:14 GMT

sh1

sh2

Re: Search head and Indexer has different indexes list

moonyoungjung — Thu, 29 Aug 2019 07:22:18 GMT

Thank you for your answer.

I was checked entire 20 ea indexer's directory //splunk_home/var/lib/splunk/
Every directory has same folder structure. and same index folder name
and I was checked setting of SH1, SH2, ...SH5 Distributed search » Search peers in the list.
The list is the same.

What conf file should I fix or check?

Re: Search head and Indexer has different indexes list

inventsekar — Thu, 29 Aug 2019 09:58:33 GMT

Some "search head " has indexes 10 list of it
the other has 15 list in setting -> indexes

-----this part is bit confusing.. could you please write your question once again. thanks.

Re: Search head and Indexer has different indexes list

moonyoungjung — Wed, 30 Sep 2020 01:57:27 GMT

I will Explained more detail.
Sh1 Settings -> data -> indexes list (23 index)
Sh2 Settings -> data -> indexes list (20 index)
Is different.
And entire indexes is 40 every index cluster members has 40 indexes.

Sh1 has 17 index it's not sync with indexer cluster
and Sh2 has 20 it's not sync with indexer cluster too.

This situation is problem when I was add data from sh1 and sh2
for example I want add data to idx_a but there is no idx_a in sh1 when select index infomation step.

Re: Search head and Indexer has different indexes list

moonyoungjung — Wed, 30 Sep 2020 02:01:06 GMT

I will explain more detail.
sh1 settings->data->indexes list(23 index)
sh1 settings->data->indexes list(20 index)
is different.
And entire indexes is 40.
Every index cluster members have 40 indexes list of it.

sh1 is 17 index not sync with entire index list
sh2 is 20 index not sync with entire index list too.

This situation is a problem when I added data from sh1 and sh2.
For example, I want to add data to idx_a but there is no idx_a in sh1 when select index information step.