All Apps and Add-ons

Why am I unable to retrieve events when searching with index=* ?

Atchyuth_P
Path Finder

Hi Team,

Environment

1 - Search Head, 2-Indexers, 1 - Deployment Server, 1 - Heavy Forwarder, 1 -Cluster Master

Problem Statement

1)I am unable to retrieve events when searching with index=* 

  2) When checked with connectives all were connected (SH --> Indexers --> CM --> HF --> DS)

When checked with internal index showing 401 client is not authenticated.

Atchyuth_P_0-1671089996297.png

When checked from backend there is no error showing in splunkd.log

 

 

Tags (1)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @Atchyuth_P,

wher do you runned the search with results?

if you see data in HF, there something wrong in your configuration because there are two choices:

  • you have a local copy of data,
  • you configured your HF as SH,

in both cases it isn't correct.

As I said: where do you runned the search with 0 results?

If in Indexer, it's correct because you cannot use Indexers for searching only SH.

If in SH you have to debug: are other searches running on SH (e.g. index=_internal)?

Configurations seems to be ok.

Ciao.

Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Atchyuth_P,

are you speaking of searches on SH or on IDXs?

if you have an IDXs Cluster, you cannot use them for searching only SH.

The other systems cannot be used for searching, only SH.

for using other systems for searching, you have to configurate them as SH.

Ciao.

Giuseppe

0 Karma

Atchyuth_P
Path Finder

Hi @gcusello 

 

Ok, i found the mistake that i have done but from HF the data is not pushing to indexers.

I am sharing the screenshots for reference

Heavy Forwarder : 

inputs.conf

Atchyuth_P_0-1671116120300.png

outputs.conf

Atchyuth_P_1-1671116509716.png

Indexer 1

inputs.conf

Atchyuth_P_2-1671116571052.png

Atchyuth_P_4-1671116708988.png

 

Indexer 2

Atchyuth_P_3-1671116631353.png

Atchyuth_P_4-1671116708988.png

When i check with connectivity all were connected

The index is showing "0" Events

Atchyuth_P_5-1671116952618.png

In HF i can see the data

Atchyuth_P_6-1671116984116.png

 

Please suggest

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Atchyuth_P,

wher do you runned the search with results?

if you see data in HF, there something wrong in your configuration because there are two choices:

  • you have a local copy of data,
  • you configured your HF as SH,

in both cases it isn't correct.

As I said: where do you runned the search with 0 results?

If in Indexer, it's correct because you cannot use Indexers for searching only SH.

If in SH you have to debug: are other searches running on SH (e.g. index=_internal)?

Configurations seems to be ok.

Ciao.

Giuseppe

0 Karma

Atchyuth_P
Path Finder

Hi @gcusello 

Atchyuth_P_0-1671118405855.png

I can see for HF to Indexer 2 the connection is in TIME_WAIT and for indexer 1 it is established

Yes there is a local copy but when i tried to check previously it worked the events got shown in indexer 2 but not in indexer 1

Now the data is not showing in two indexers

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Atchyuth_P,

outputs.conf seems to be corrects, did you checked the connection between HF and IDX2 (if not try using telnet not ping)?

About local copy you shouldn't have it also because you have in your outputs.conf  "indexAndForward = false"

I repeat the question: where are you running searches: on SH or on another system?

How do you configured SH to search on IDXs?

Ciao.

Giuseppe

0 Karma

Atchyuth_P
Path Finder

Hi @gcusello 

 

I am trying to check the search in both the indexers because the events is showing zero

Atchyuth_P_1-1671120331674.png

I tried both telnet and ping HF---> IDX2,IDX2 ---> HF all the connection established

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Atchyuth_P,

ping isn't relevant to check connections, uso only telnet on port 9997.

About searches: you cannot use Indexers (when clustered) for searching only Search Heads.

If search runs on a IDX means that there's a misconfiguration in the cluster.

What does it happen running a search a different index (obviously on SH)?

Ciao.

Giuseppe

 

0 Karma

Atchyuth_P
Path Finder

Hi @gcusello

 

Thanks for the info i miss the catch i have done the configuration in SH as well. Almost, forgot IDX will not acts as SH.

Sorry for the trouble. 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Atchyuth_P,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Atchyuth_P,

no problem, tell me if I can help you more on this issue, otherwise, if one answer solves your need, please accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the Contributors;-)

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...