cant search data in indexer

[arsidiq]

i installed splunk in distributed management environment. furthermore, my indexer server got reboot and i can't query my data even though at index = _internal. whereas previously it was fine.

[PickleRick]

First things first.

1. Does the splunkd process run on the indexer?

2. Does it listen on the 8089 port?

3. Can you reach indexer's 8089 port from the SH?

4. What does "splunk status" say on the indexer?

[isoutamo]

Can you tell more about what and how you have done this installation and what kind of distributed environment you have?

Are the problematic node indexer, search head or something other node?

[arsidiq]

first i have 3 different server (HF, SH, and IDX) and the distributed search is going to IDX. there an incident that idx server is shutting down and after i started and run the splunk services, i can't query any data. i try to query index = * and has no result.

[arsidiq]

i think the problem itself in indexer node, but still cant find out why it can query splunk internal log

[isoutamo]

Have you check that those indexes are there and splunk is running there without issues?

Basically if you have GUI enabled on IDX you can try query from there or use CLI and do queries on command line too.

Check also if there is any issues with internal logs. You can query those from internal indexes like 

index=_internal log_level IN (error, warn)

[kiran_panchavat]

@arsidiq 

Refer this 

Solved: Why is no data being written to the _internal inde... - Splunk Community

Solved: Why is _internal index is disabled? - Splunk Community

[kiran_panchavat]

@arsidiq 

Verify that the search head can communicate with the indexer.

If it fails, check firewall rules or network issues. Ensure the indexer is listed in the search head’s distributed search configuration:

• Splunk Web: Settings > Distributed Search > Search Peers.
• Or check $SPLUNK_HOME/etc/system/local/distsearch.conf.

Check this on the indexer:-  tail -n 100 /opt/splunk/var/log/splunk/splunkd.log

[kiran_panchavat]

@arsidiq 

• Ensure the indexer is running. Log into the indexer server and check Splunk's status: /opt/splunk/bin/splunk status
• If Splunk is not running, start it: /opt/splunk/bin/splunk start

• Confirm that the search head and other components can communicate with the indexer. Test connectivity using: ping <indexer_ip>
• Verify that the Splunk management port (default: 8089) is open: telnet <indexer_ip> 8089
  Check the Splunk logs on the indexer for errors: /opt/splunk/var/log/splunk/splunkd.log
• Look for issues related to indexing, disk space, or corrupted buckets. Common issues include: Disk full errors or Corrupted index buckets due to improper shutdown.

[arsidiq]

yups the indexer is running, and still cant quey any data after the server has been reboot

[kiran_panchavat]

@arsidiq 

Verify permissions for Splunk directories. If they've changed to root after a reboot, correct them with:

chown -R splunk:splunk /opt/splunk

Are you able to see the data for other indexes? 

[arsidiq]

already done this, since splunk has to run using user splunk sir so when i want to start the service i already change the permissions