Splunk Search

Role/User default search indexes not working

yaleman
Engager

I can't tell if it's since we moved to mounted bundles or not, but recently we need to explicitly set the indexes which we wish to search. It was working perfectly not long ago. Doing a search of just * gives not much at all - only main, _internal, _audit, _introspection and sos. There's another ~15 indexes with MANY more events in them on our system.

I've gone as far as clicking "add all" on the "default indexes searched" box and it doesn't seem to make any difference. This is breaking all our apps, amongst other issues.

We've got a single search head (v6.1.x) pointing at a single indexer (v6.0.x) both running Debian with the indexer NFS-mounting the search head's /opt/splunk/etc/ directory directly.

Tags (1)
0 Karma

Lucas_K
Motivator

Check distributed search on the search head (settings/distributed search/search peers). Do you see your index listed correctly as "UP"?

Do you have your mount point correctly configured on the indexer? The incoming search from the sh to the idx needs this so it can correctly get app configurations.

on your indexer check your distsearch.conf - /opt/splunk/etc/system/local/distsearch.conf

[my_searchhead]
mounted_bundles=true
bundles_location=/mnt/shared_bundle

Check your auth on the search head.
You can use btool to show this.
splunk btool authorize list --debug

Find your role does it have the correct indexes listed?

If not fix them in /opt/splunk/etc/system/local/authorize.conf on your search head.

0 Karma

Lucas_K
Motivator

Did the btool output look the same as what your expecting to see?

My comment about the bundle was that you've exported /opt/splunk/etc directly. Nothing in the docs said to do that, it should work as the indexer should still be able to figure out where the app artifacts are. I don't 100% know if it picks up anything else that it shouldn't be seeing. My mounts consist of apps/pooling/system and users dirs only nothing else (but that is shared storage not the search heads etc dir).

0 Karma

yaleman
Engager

Apologies, yes. The index is showing status of "up" and replication status of "mounted". Searches work fine (with no errors) as long as we specify the index manually in the search string.

I'm not sure what you mean by the fact that it's abnormal - it's configured exactly as per the docs?

0 Karma

Lucas_K
Motivator

You didn't mention if you can see the index from the search head (settings/distributed search/search peers).

Also, tried switching back to non-mounted bundles? Your bundling setup seems abnormal compared to the documentation.

0 Karma

yaleman
Engager

The distsearch.conf seems to be OK, matches up with the mounted directory. My account's got two roles (admin and can_delete) - which it always has had.

The authorize.conf seems sane - admin has a long list of srchIndexesDefault (including *) - and can_delete doesn't have a line for that. I'm stumped.

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...