SPL2 can access only a subset of available indexes, while SPL1 can access all of them

[vnetrebko]

Hi everyone,

I’m testing SPL2 for the first time after upgrading our Splunk deployment from 10.0 to 10.2.1, and I’m seeing behavior that looks very different from SPL1.

When using index=* | stats count by index in spl1 we can access all the indexes available on our indexer-cluster. 
Using similar query with spl2 returns only 13. All the indexes that spl2 returns appears as "internal", so it's ES indexes and all "_*" indexes.

Trying to search available in spl1 index, for example: index=aws results in error:
Error in 'SearchParser': The SPL2 query is invalid: 'Unable to find any index datasets matching search expression: 'index="aws"'. It's possible that they exist, but you are not authorized to access them. Contact your administrator.'.

We are using single Search Head with Indexer Cluster.

[richgalloway]

Do you have the same indexes installed on both the indexer cluster and the search head?  Not sure it matters, but it's possible SPL2 is trying to validate index names.

[vnetrebko]

We don't have the same set of indexes on SH, we actually try to avoid having indexes.conf on our SH in apps we deploy/develop.
All indexes definitions are stored on Indexer Cluster, except for cases where we need it for app functionality. I always thought it's the best practice not to store them on SH.

[PickleRick]

Having indexes.conf on other components is actually a good practice. It allows you to have working completion in SPL editor dialog and allows for working index choice dialogs on HFs (for some inputs it's mandatory to have those entries, otherwise you're not able to create inputs since you can't choose "non-existing" indexes in GUI and can't save inputs.conf stanza without properly validating the input in GUI).

[richgalloway]

Consider adding indexes to the SH.  SPL2 may be another instance of app functionality that needs it.