Hello
I want to monitor the health of db connect app inputs and connections and i noticed the the health monitor is not working. im getting the message "search populated no results"
When i tried to investigate the issue i found out that index=_internal is empty
I guess its related.
Can you please help me figure out why the index is empty and the health monitor is not working ?
If a search "index=_internal" over the last 24 hours is empty, I can think of a couple of reasons.
Most likely - your role doesn't have administrative access. (More specifically, it doesn't have access to the _internal index, which is usually limited to admins). Either log in as an administrator with access to _internal, or have your Splunk folks add this index to your role.
It's also possible that you have DBX installed on a heavy forwarder. That HF has been told its outputs need to go to your real indexer(s), but it's never been told to *search* the indexer when someone searches for "index=_internal". The steps you might need are https://docs.splunk.com/Documentation/Splunk/9.2.0/DistSearch/Configuredistributedsearch#Use_Splunk_...
Anyway, if you can confirm the above two things, either one of them is the issue, or you can report back here with what you've found!
-Rich