Why is _internal index is disabled?

[rashi83]

I want to enable the _internal index but cant from UI.

Authorize.conf (etc\search\local) is also attached and has _internal index added in srchIndexesDefault for both Admin and User roles.

I have admin role.

[role_can_delete]
deleteIndexesAllowed = *;main
indexes_edit = enabled
srchMaxTime = 8640000

[role_power]
rtsearch = disabled
schedule_rtsearch = disabled
srchMaxTime = 8640000
srchTimeWin = 0

[role_user-marttestuser]
cumulativeRTSrchJobsQuota = 0
cumulativeSrchJobsQuota = 0
rtSrchJobsQuota = 0
srchDiskQuota = 0
srchIndexesAllowed = *
srchIndexesDefault = main
srchJobsQuota = 0
srchMaxTime = 0

[role_admin]
admin_all_objects = disabled
edit_modinput_web_ping = disabled
grantableRoles = admin
importRoles = can_delete;power;user
indexes_edit = disabled
schedule_rtsearch = disabled
srchIndexesAllowed = *;gb_bed_app_dev;gb_bed_app_qa;gb_bed_iis_dev;gb_bed_iis_prod;gb_bed_iis_qa;main;main_uat;prod_sharepoint_iis;tam;uat_sharepoint_iis
srchIndexesDefault = *;_*;aem;aem_test;apac;appserver;aws_bdl_dev;aws_haproxy_dev;aws_haproxy_prd;aws_haproxy_stg;casd_2018;gb_bed_app_dev;gb_bed_app_et2;gb_bed_app_log;gb_bed_app_prod;gb_bed_app_qa;gb_bed_app_uat;gb_bed_ihs_uat;gb_bed_iis_dev;gb_bed_iis_prod;gb_bed_iis_qa;gb_bed_iis_uat;gb_exe_app_et2_uat;haproxy_test;jenkins;jenkins_artifact;jenkins_console;jenkins_statistics;la_fkl_app_dev_ohs;la_fkl_app_prod_ohs;main;main_uat;marsh-aws-dev;marsh_aws_apigee_dev;marsh_aws_apigee_prod;marsh_aws_apigee_stg;marsh_aws_dev;marsh_aws_prd;marsh_aws_stg;marsh_aws_tableau_dev;marsh_onprem_dev_qa;marsh_onprem_dr;marsh_onprem_perf_uat;marsh_onprem_prod;ping;prod_sharepoint_iis;summary;tam;uat_sharepoint_iis;us_dfw_apigee_dev;us_dfw_apigee_prod;us_dfw_apigee_stg;us_dfw_app_prod_access;us_dfw_app_prod_apigee;us_dfw_dbint_prod_postgres;us_dfw_etl_prod;us_dfw_prod_app_csvaudit;us_dfw_prod_webext_ping;us_dfw_prod_webint_ping;us_dfw_springboot_stg;us_dfw_webext_aws_dev;us_dfw_webext_aws_prod;us_dfw_webext_dev_devops;us_dfw_webext_prod_devops;us_dfw_webext_prod_tam;us_dfw_webext_prod_temp;us_dfw_webext_stg_devops;us_dfw_webint_dev_devops;us_dfw_webint_prod;us_dfw_webint_prod_access;us_dfw_webint_prod_devops;us_dfw_webint_prod_haproxy;us_dfw_webint_prod_tam;us_micredit_cloud_web_prod_iis;webserver;wineventlog;_internal
srchMaxTime = 8640000

[role_user]
admin_all_objects = enabled
rtsearch = enabled
srchIndexesAllowed = *;gb_bed_app_dev;gb_bed_app_et2;gb_bed_app_log;gb_bed_app_qa;gb_bed_iis_dev;gb_bed_iis_prod;gb_bed_iis_qa;gb_bed_iis_uat;main;prod_sharepoint_iis;tam;uat_sharepoint_iis
srchIndexesDefault = *;gb_bed_app_dev;gb_bed_app_et2;gb_bed_app_log;gb_bed_app_qa;gb_bed_iis_dev;gb_bed_iis_prod;gb_bed_iis_qa;gb_bed_iis_uat;main;prod_sharepoint_iis;tam;uat_sharepoint_iis;us_dfw_prod_velocity_log4j;us_dfw_uat_velocity_log4j;us_urb_prod_web_access;_internal
srchMaxTime = 8640000

[jkat54 · Accepted solution]

srchIndexesAllowed doesn't have your index you're looking for. You only have it under srchIndexesDefault unless the forum treated some of your paste as markup.

[rashi83]

I have added capability to Edit Index to my role and also added _internal index name in defaultsearch and searchindexes allowed.

_internal index is now searchable.

Thank you all

[jkat54]

Can you accept my answer if it helped?

[mdsnmss]

Has anything on the system recently been restored from backup? Was _internal disabled manually or by Splunk itself? You can look for something like this in splunkd.log that can help tell you where the problem is:

06-20-2019 23:06:05.513 +0000 ERROR IndexProcessor - One or more indexes could not be initialized and were automatically disabled, please see splunkd.log for more details

06-20-2019 23:06:05.512 +0000 ERROR IndexProcessor - caught exception for index=_internal during initialzation: 'Splunk has detected that a directory has been manually copied into its database, causing id conflicts [/opt/splunk/var/lib/splunk/_internaldb/db/hot_v1_1, /opt/splunk/var/lib/splunk/_internaldb/db/db_1307563383_1307490866_1].'.Disabling the index, please fix-up and run splunk enable index

If this is the case, Splunk would need to be stopped, bucket collisions remedied, indexes reenabled, and then restarted. A good answer on this can be found here: https://answers.splunk.com/answers/30986/why-is-my-index-disabled.html.

[nvanderwalt_spl]

Well, that is indexed in the _internal index, which is disabled. May have to use grep on the filesystem there...

[jkat54]

grep ERROR $SPLUNK_HOME/var/log/splunk/splunkd.log