Why am I unable to retrieve events when searching with index=* ?

[Atchyuth_P]

Hi Team,

Environment

1 - Search Head, 2-Indexers, 1 - Deployment Server, 1 - Heavy Forwarder, 1 -Cluster Master

Problem Statement

1)I am unable to retrieve events when searching with index=* 

  2) When checked with connectives all were connected (SH --> Indexers --> CM --> HF --> DS)

When checked with internal index showing 401 client is not authenticated.

When checked from backend there is no error showing in splunkd.log

[gcusello]

Hi @Atchyuth_P,

are you speaking of searches on SH or on IDXs?

if you have an IDXs Cluster, you cannot use them for searching only SH.

The other systems cannot be used for searching, only SH.

for using other systems for searching, you have to configurate them as SH.

Ciao.

Giuseppe

[Atchyuth_P]

Hi @gcusello 

Ok, i found the mistake that i have done but from HF the data is not pushing to indexers.

I am sharing the screenshots for reference

Heavy Forwarder : 

inputs.conf

outputs.conf

Indexer 1

inputs.conf

Indexer 2

When i check with connectivity all were connected

The index is showing "0" Events

In HF i can see the data

Please suggest

[gcusello · Accepted solution]

Hi @Atchyuth_P,

wher do you runned the search with results?

if you see data in HF, there something wrong in your configuration because there are two choices:

• you have a local copy of data,
• you configured your HF as SH,

in both cases it isn't correct.

As I said: where do you runned the search with 0 results?

If in Indexer, it's correct because you cannot use Indexers for searching only SH.

If in SH you have to debug: are other searches running on SH (e.g. index=_internal)?

Configurations seems to be ok.

Ciao.

Giuseppe

[Atchyuth_P]

Hi @gcusello 

I can see for HF to Indexer 2 the connection is in TIME_WAIT and for indexer 1 it is established

Yes there is a local copy but when i tried to check previously it worked the events got shown in indexer 2 but not in indexer 1

Now the data is not showing in two indexers

[gcusello]

Hi @Atchyuth_P,

outputs.conf seems to be corrects, did you checked the connection between HF and IDX2 (if not try using telnet not ping)?

About local copy you shouldn't have it also because you have in your outputs.conf  "indexAndForward = false"

I repeat the question: where are you running searches: on SH or on another system?

How do you configured SH to search on IDXs?

Ciao.

Giuseppe

[Atchyuth_P]

Hi @gcusello 

I am trying to check the search in both the indexers because the events is showing zero

I tried both telnet and ping HF---> IDX2,IDX2 ---> HF all the connection established

[gcusello]

Hi @Atchyuth_P,

ping isn't relevant to check connections, uso only telnet on port 9997.

About searches: you cannot use Indexers (when clustered) for searching only Search Heads.

If search runs on a IDX means that there's a misconfiguration in the cluster.

What does it happen running a search a different index (obviously on SH)?

Ciao.

Giuseppe

[Atchyuth_P]

Hi @gcusello

Thanks for the info i miss the catch i have done the configuration in SH as well. Almost, forgot IDX will not acts as SH.

Sorry for the trouble. 

[gcusello]

Hi @Atchyuth_P,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

[gcusello]

Hi @Atchyuth_P,

no problem, tell me if I can help you more on this issue, otherwise, if one answer solves your need, please accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the Contributors;-)