topic Re: cant search data in indexer in Installation

cant search data in indexer

arsidiq — Thu, 24 Apr 2025 08:49:08 GMT

i installed splunk in distributed management environment. furthermore, my indexer server got reboot and i can't query my data even though at index = _internal. whereas previously it was fine.

Re: cant search data in indexer

kiran_panchavat — Thu, 24 Apr 2025 09:07:18 GMT

@arsidiq

Ensure the indexer is running. Log into the indexer server and check Splunk's status: /opt/splunk/bin/splunk status
If Splunk is not running, start it: /opt/splunk/bin/splunk start

Confirm that the search head and other components can communicate with the indexer. Test connectivity using: ping <indexer_ip>
Verify that the Splunk management port (default: 8089) is open: telnet <indexer_ip> 8089
Check the Splunk logs on the indexer for errors: /opt/splunk/var/log/splunk/splunkd.log
Look for issues related to indexing, disk space, or corrupted buckets. Common issues include: Disk full errors or Corrupted index buckets due to improper shutdown.

Re: cant search data in indexer

arsidiq — Thu, 24 Apr 2025 09:07:22 GMT

yups the indexer is running, and still cant quey any data after the server has been reboot

Re: cant search data in indexer

kiran_panchavat — Thu, 24 Apr 2025 09:12:28 GMT

@arsidiq

Verify that the search head can communicate with the indexer.

If it fails, check firewall rules or network issues. Ensure the indexer is listed in the search head’s distributed search configuration:

Splunk Web: Settings > Distributed Search > Search Peers.
Or check $SPLUNK_HOME/etc/system/local/distsearch.conf.

Check this on the indexer:- tail -n 100 /opt/splunk/var/log/splunk/splunkd.log

Re: cant search data in indexer

kiran_panchavat — Thu, 24 Apr 2025 09:16:46 GMT

@arsidiq

Refer this

Solved: Why is no data being written to the _internal inde... - Splunk Community

Solved: Why is _internal index is disabled? - Splunk Community

Re: cant search data in indexer

kiran_panchavat — Thu, 24 Apr 2025 09:21:51 GMT

@arsidiq

Verify permissions for Splunk directories. If they've changed to root after a reboot, correct them with:

chown -R splunk:splunk /opt/splunk

Are you able to see the data for other indexes?

Re: cant search data in indexer

isoutamo — Thu, 24 Apr 2025 09:23:05 GMT

Can you tell more about what and how you have done this installation and what kind of distributed environment you have?

Are the problematic node indexer, search head or something other node?

Re: cant search data in indexer

arsidiq — Thu, 24 Apr 2025 09:35:51 GMT

already done this, since splunk has to run using user splunk sir so when i want to start the service i already change the permissions

Re: cant search data in indexer

arsidiq — Thu, 24 Apr 2025 09:38:57 GMT

first i have 3 different server (HF, SH, and IDX) and the distributed search is going to IDX. there an incident that idx server is shutting down and after i started and run the splunk services, i can't query any data. i try to query index = * and has no result.

Re: cant search data in indexer

arsidiq — Thu, 24 Apr 2025 09:40:09 GMT

i think the problem itself in indexer node, but still cant find out why it can query splunk internal log

Re: cant search data in indexer

isoutamo — Mon, 28 Apr 2025 17:31:33 GMT

Have you check that those indexes are there and splunk is running there without issues?

Basically if you have GUI enabled on IDX you can try query from there or use CLI and do queries on command line too.

Check also if there is any issues with internal logs. You can query those from internal indexes like

index=_internal log_level IN (error, warn)

Re: cant search data in indexer

PickleRick — Mon, 28 Apr 2025 20:29:53 GMT

First things first.

1. Does the splunkd process run on the indexer?

2. Does it listen on the 8089 port?

3. Can you reach indexer's 8089 port from the SH?

4. What does "splunk status" say on the indexer?