topic Re: Splunk Troubleshhoting in All Apps and Add-ons

Why am I unable to retrieve events when searching with index=* ?

Atchyuth_P — Thu, 15 Dec 2022 09:53:18 GMT

Hi Team,

Environment

1 - Search Head, 2-Indexers, 1 - Deployment Server, 1 - Heavy Forwarder, 1 -Cluster Master

Problem Statement

1)I am unable to retrieve events when searching with index=*

2) When checked with connectives all were connected (SH --> Indexers --> CM --> HF --> DS)

When checked with internal index showing 401 client is not authenticated.

When checked from backend there is no error showing in splunkd.log

Re: Splunk Troubleshhoting

gcusello — Thu, 15 Dec 2022 07:54:53 GMT

Hi @Atchyuth_P,

are you speaking of searches on SH or on IDXs?

if you have an IDXs Cluster, you cannot use them for searching only SH.

The other systems cannot be used for searching, only SH.

for using other systems for searching, you have to configurate them as SH.

Ciao.

Giuseppe

Re: Splunk Troubleshhoting

Atchyuth_P — Thu, 15 Dec 2022 15:11:26 GMT

Hi @gcusello

Ok, i found the mistake that i have done but from HF the data is not pushing to indexers.

I am sharing the screenshots for reference

Heavy Forwarder :

inputs.conf

outputs.conf

Indexer 1

inputs.conf

Indexer 2

When i check with connectivity all were connected

The index is showing "0" Events

In HF i can see the data

Please suggest

Re: Splunk Troubleshhoting

gcusello — Thu, 15 Dec 2022 15:29:35 GMT

Hi @Atchyuth_P,

wher do you runned the search with results?

if you see data in HF, there something wrong in your configuration because there are two choices:

you have a local copy of data,
you configured your HF as SH,

in both cases it isn't correct.

As I said: where do you runned the search with 0 results?

If in Indexer, it's correct because you cannot use Indexers for searching only SH.

If in SH you have to debug: are other searches running on SH (e.g. index=_internal)?

Configurations seems to be ok.

Ciao.

Giuseppe

Re: Splunk Troubleshhoting

Atchyuth_P — Thu, 15 Dec 2022 15:37:55 GMT

Hi @gcusello

I can see for HF to Indexer 2 the connection is in TIME_WAIT and for indexer 1 it is established

Yes there is a local copy but when i tried to check previously it worked the events got shown in indexer 2 but not in indexer 1

Now the data is not showing in two indexers

Re: Splunk Troubleshhoting

gcusello — Thu, 15 Dec 2022 15:47:50 GMT

Hi @Atchyuth_P,

outputs.conf seems to be corrects, did you checked the connection between HF and IDX2 (if not try using telnet not ping)?

About local copy you shouldn't have it also because you have in your outputs.conf "indexAndForward = false"

I repeat the question: where are you running searches: on SH or on another system?

How do you configured SH to search on IDXs?

Ciao.

Giuseppe

Re: Splunk Troubleshhoting

Atchyuth_P — Thu, 15 Dec 2022 16:06:33 GMT

Hi @gcusello

I am trying to check the search in both the indexers because the events is showing zero

I tried both telnet and ping HF---> IDX2,IDX2 ---> HF all the connection established

Re: Splunk Troubleshhoting

gcusello — Thu, 15 Dec 2022 16:19:45 GMT

Hi @Atchyuth_P,

ping isn't relevant to check connections, uso only telnet on port 9997.

About searches: you cannot use Indexers (when clustered) for searching only Search Heads.

If search runs on a IDX means that there's a misconfiguration in the cluster.

What does it happen running a search a different index (obviously on SH)?

Ciao.

Giuseppe

Re: Splunk Troubleshhoting

Atchyuth_P — Thu, 15 Dec 2022 16:56:10 GMT

Hi @gcusello

Thanks for the info i miss the catch i have done the configuration in SH as well. Almost, forgot IDX will not acts as SH.

Sorry for the trouble.

Re: Splunk Troubleshhoting

gcusello — Fri, 16 Dec 2022 07:35:03 GMT

Hi @Atchyuth_P,

no problem, tell me if I can help you more on this issue, otherwise, if one answer solves your need, please accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the Contributors;-)

Re: Splunk Troubleshhoting

gcusello — Fri, 16 Dec 2022 13:59:32 GMT

Hi @Atchyuth_P,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉