topic Re: Introspection in Splunk Search

Introspection

SN1 — Tue, 25 Mar 2025 05:04:47 GMT

Hello I am running search
index=_introspection
dedup host
table host

in result i am not able to see one indexer and one search head while other indexers and sh are visible .

Re: Introspection

kiran_panchavat — Tue, 25 Mar 2025 06:06:58 GMT

@SN1

Check if the missing indexer and search head are online and Splunk is running on them. You can SSH into those servers and run splunk status to verify.

Are you able to see all the instances in the Monitoring console?

This could happen if:

The hosts are down or disconnected.
The Splunk instance on those hosts is not running.
There’s a network issue preventing data from being forwarded.

Re: Introspection

SN1 — Tue, 25 Mar 2025 06:10:54 GMT

i am getting this error on health check

Root Cause(s):
- Events from tracker.log have not been seen for the last 238401 seconds, which is more than the red threshold (210 seconds). This typically occurs when indexing or forwarding are falling behind or are blocked.

Re: Introspection

kiran_panchavat — Tue, 25 Mar 2025 06:11:02 GMT

@SN1

If you run this search, how many peers return count?

index=_internal earliest=-5m@m | stats count by splunk_server

This should give responses from all your indexers, and if you have your SH / Component boxes configured to forward their internal logs, those also.

Re: Introspection

kiran_panchavat — Tue, 25 Mar 2025 06:17:51 GMT

@SN1

The _introspection index in Splunk is part of the "Platform Instrumentation" features, which collect information about your systems running Splunk to help diagnose performance issues.

What does platform instrumentation log? - Splunk Documentation

Introspection endpoint descriptions - Splunk Documentation

Re: Introspection

kiran_panchavat — Tue, 25 Mar 2025 06:40:27 GMT

@SN1

There should be a message in splunkd.log explaining the problem.

index=_internal source=*splunkd.log

Check that there is enough storage on the volume containing the introspection index.

Also, confirm no one turned off introspection. See

https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/ConfigurePIF#Disable_logging

If the missing hosts haven’t reported data recently, they might not appear depending on the default time range (e.g., last 24 hours). Expand the time range in the UI or add earliest=-30d (or further back) to your search

Re: Introspection

livehybrid — Tue, 25 Mar 2025 07:21:14 GMT

Hi @SN1

If you look further back, when was the last event?
Have a look using this search looking back at least to the time of the last event from the missing servers.

| tstats latest(_time) as _time where index=_introspection by host

Then run the search 5-10 minutes later. Are the times of the last events different for the missing host? If so this would suggest that they are having issues sending logs and that they are delayed, rather than not sending at all.

In addition it would be worth checking the Splunk log of the missing host directly, check out $SPLUNK_HOME/var/log/splunk/splunkd.log - are there any references to blocking or output errors?

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will