Deployment Architecture

No _internal results from distributed search head

_smp_
Builder

As a pretty new user, I recently installed the Universal Forwarder on a Linux server, created a file input, and forwarded to an indexer. This was working fine. Then as a result of a support case, I had to change the role from a UF to a Search Head in Distributed Search. After doing this and configuring the SH to forward its logs to the indexer, I am unable to return any results with a simple index=_internal search. Yet I can get results from all the non-internal indexes just fine. I have another SH (non-clustered) that works, and I have closely compared the Roles, but found no differences.

After searching the forum, I found a number of references to outputs.conf - here's mine:

[indexAndForward]
index = false

[tcpout]
defaultGroup = indexer
forwardedindex.filter.disable = true
indexAndForward = false

Not sure what else to look for?

0 Karma
1 Solution

lguinn2
Legend

Did you also set the search head to search the indexers?

By default, only a user with Admin privileges can see the internal indexes (_internal, _audit, etc.)
Check the role and make sure that these indexes are allowed to be searched.

View solution in original post

lguinn2
Legend

Did you also set the search head to search the indexers?

By default, only a user with Admin privileges can see the internal indexes (_internal, _audit, etc.)
Check the role and make sure that these indexes are allowed to be searched.

_smp_
Builder

The admin role was the problem. On my SH that was working, the admin role was restricted to internal and non-internal indexes. The SH which was broke, there were no selected indexes. After I added both internal and non-internal indexes to the admin role, I get the search results I was expecting.

Thank you.

This dialog box is confusing to me though. What is the default for the admin role - internal/non-internal, or nothing? The description for the dialog box explicitly states "Restrict", so my assumption is that if no indexes are selected, then there no restrictions. Am I wrong about this?

0 Karma

lguinn2
Legend

On the role, there are two index settings: indexes that the role is allowed to see ("Indexes" at the very bottom) and indexes that are default.

I can see how you might be confused by the language, but your assumption is wrong.
ONLY the indexes that appear in the "Indexes" list can be searched. Either an index must be explicitly chosen, or you can choose "All internal indexes" or "All non-internal indexes".

The "Indexes searched by default" must be a subset of that final "Indexes" list.

For the admin role, I usually set it like this -
Indexes searched by default = All non-internal indexes
Indexes = All non-internal indexes AND All internal indexes

_smp_
Builder

Thank you for your clarification.

0 Karma
Get Updates on the Splunk Community!

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...