All Splunk internal indexes were disabled with red lock icons

[coweatgrass14]

Dear All,

All of the internal indexes of Splunk, (_audit, _internal, _introspection, _metrics, _telemetry, _thefishbucket and splunklogger) were disabled with  red lock icons.

I have tried:

1) restart the splunkd;

2) followed the method in the following link (delete the entire _audit folder but no luck)
URL: https://community.splunk.com/t5/Archive/audit-index-remains-disabled/m-p/98864

Please help me.  Thank you.

[coweatgrass14]

I found the only one system index ( summary is working, others are disabled )

 

[coweatgrass14]

/opt/splunk/etc/system/local/indexes.conf

(I tried to rename the indexes.conf to o_indexes.conf and restart the splunk )

In addition, I try to upgrade to latest version of splunk and still no luck. <>_<>

[_audit]
archiver.enableDataArchive = 0
bucketRebuildMemoryHint = 0
compressRawdata = 1
enableDataIntegrityControl = 0
enableOnlineBucketRepair = 1
enableTsidxReduction = 0
metric.enableFloatingPointCompression = 1
minHotIdleSecsBeforeForceRoll = 0
rtRouterQueueSize =
rtRouterThreads =
selfStorageThreads =
suspendHotRollByDeleteQuery = 0
syncMeta = 1
tsidxWritingLevel =

/opt/splunk/etc/system/default/indexes.conf contents:
( this file last modified havent been modified since 2 years ago)

1 # Version 8.0.0
2 # DO NOT EDIT THIS FILE!
3 # Changes to default files will be lost on update and are difficult to
4 # manage and support.
5 #
6 # Please make any changes to system defaults by overriding them in
7 # apps or $SPLUNK_HOME/etc/system/local
8 # (See "Configuration file precedence" in the web documentation).
9 #
10 # To override a specific setting, copy the name of the stanza and
11 # setting to the file where you wish to override it.
12 #
13 # This file configures Splunk's indexes and their properties.
14 #
15
16 ################################################################################
17 # "global" params (not specific to individual indexes)
18 ################################################################################
19 sync = 0
20 indexThreads = auto
21 memPoolMB = auto
22 defaultDatabase = main
23 enableRealtimeSearch = true
24 suppressBannerList =
25 maxRunningProcessGroups = 8
26 maxRunningProcessGroupsLowPriority = 1
27 bucketRebuildMemoryHint = auto
28 serviceOnlyAsNeeded = true
29 serviceSubtaskTimingPeriod = 30
30 serviceInactiveIndexesPeriod = 60
31 maxBucketSizeCacheEntries = 0
32 processTrackerServiceInterval = 1
33 hotBucketTimeRefreshInterval = 10
34 rtRouterThreads = 0
35 rtRouterQueueSize = 10000
36 selfStorageThreads = 2
37 fileSystemExecutorWorkers = 5
38
39 ################################################################################
40 # index specific defaults
41 ################################################################################
42 maxDataSize = auto
43 maxWarmDBCount = 300
44 frozenTimePeriodInSecs = 188697600
45 rotatePeriodInSecs = 60
46 coldToFrozenScript =
47 coldToFrozenDir =
48 compressRawdata = true
49 maxTotalDataSizeMB = 500000
50 maxGlobalRawDataSizeMB = 0
51 maxGlobalDataSizeMB = 0
52 maxMemMB = 5
53 maxConcurrentOptimizes = 6
54 maxHotSpanSecs = 7776000
55 maxHotIdleSecs = 0
56 maxHotBuckets = 3
57 minHotIdleSecsBeforeForceRoll = auto
58 quarantinePastSecs = 77760000
59 quarantineFutureSecs = 2592000
60 rawChunkSizeBytes = 131072
61 minRawFileSyncSecs = disable
62 assureUTF8 = false
63 serviceMetaPeriod = 25
64 partialServiceMetaPeriod = 0
65 throttleCheckPeriod = 15
66 syncMeta = true
67 maxMetaEntries = 1000000
68 maxBloomBackfillBucketAge = 30d
69 enableOnlineBucketRepair = true
70 enableDataIntegrityControl = false
71 maxTimeUnreplicatedWithAcks = 60
72 maxTimeUnreplicatedNoAcks = 300
73 minStreamGroupQueueSize = 2000
74 warmToColdScript=
75 tstatsHomePath = volume:_splunk_summaries/$_index_name/datamodel_summary
76 homePath.maxDataSizeMB = 0
77 coldPath.maxDataSizeMB = 0
78 streamingTargetTsidxSyncPeriodMsec = 5000
79 journalCompression = gzip
80 enableTsidxReduction = false
81 suspendHotRollByDeleteQuery = false
82 tsidxReductionCheckPeriodInSec = 600
83 timePeriodInSecBeforeTsidxReduction = 604800
84 datatype = event
85 splitByIndexKeys =
86 tsidxWritingLevel = 1
87 archiver.enableDataArchive = false
88 archiver.maxDataArchiveRetentionPeriod = 0
89 tsidxTargetSizeMB = 1500
90 metric.tsidxTargetSizeMB = 1500
91 metric.enableFloatingPointCompression = true
92 metric.compressionBlockSize = 1024
93
94 #
95 # By default none of the indexes are replicated.
96 #
97 repFactor = 0
98
99 [volume:_splunk_summaries]
100 path = $SPLUNK_DB
101
102 [provider-family:hadoop]
103 vix.mode = report
104 vix.command = $SPLUNK_HOME/bin/jars/sudobash
105 vix.command.arg.1 = $HADOOP_HOME/bin/hadoop
106 vix.command.arg.2 = jar
107 vix.command.arg.3 = $SPLUNK_HOME/bin/jars/SplunkMR-h1.jar
108 vix.command.arg.4 = com.splunk.mr.SplunkMR
109 vix.env.MAPREDUCE_USER =
110 vix.env.HADOOP_HEAPSIZE = 512
111 vix.env.HADOOP_CLIENT_OPTS = -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr
112 vix.env.HUNK_THIRDPARTY_JARS = $SPLUNK_HOME/bin/jars/thirdparty/common/avro-1.7.7.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/a
113 vix.mapred.job.reuse.jvm.num.tasks = 100
114 vix.mapred.child.java.opts = -server -Xmx512m -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr
115 vix.mapred.reduce.tasks = 0
116 vix.mapred.job.map.memory.mb = 2048
117 vix.mapred.job.reduce.memory.mb = 512
118 vix.mapred.job.queue.name = default
119 vix.mapreduce.job.jvm.numtasks = 100
120 vix.mapreduce.map.java.opts = -server -Xmx512m -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr
121 vix.mapreduce.reduce.java.opts = -server -Xmx512m -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr
122 vix.mapreduce.job.reduces = 0
123 vix.mapreduce.map.memory.mb = 2048
124 vix.mapreduce.reduce.memory.mb = 512
125 vix.mapreduce.job.queuename = default
126 vix.splunk.search.column.filter = 1
127 vix.splunk.search.mixedmode = 1
128 vix.splunk.search.debug = 0
129 vix.splunk.search.mr.maxsplits = 10000
130 vix.splunk.search.mr.minsplits = 100
131 vix.splunk.search.mr.splits.multiplier = 10
132 vix.splunk.search.mr.poll = 2000
133 vix.splunk.search.recordreader = SplunkJournalRecordReader,ValueAvroRecordReader,SimpleCSVRecordReader,SequenceFileRecordReader
134 vix.splunk.search.recordreader.avro.regex = \.avro$
135 vix.splunk.search.recordreader.csv.regex = \.([tc]sv)(?:\.(?:gz|bz2|snappy))?$
136 vix.splunk.search.recordreader.sequence.regex = \.seq$
137 vix.splunk.home.datanode = /tmp/splunk/$SPLUNK_SERVER_NAME/
138 vix.splunk.heartbeat = 1
139 vix.splunk.heartbeat.threshold = 60
140 vix.splunk.heartbeat.interval = 1000
141 vix.splunk.setup.onsearch = 1
142 vix.splunk.setup.package = current
143
144 ################################################################################
145 # index definitions
146 ################################################################################
147
148 [main]
149 homePath = $SPLUNK_DB/defaultdb/db
150 coldPath = $SPLUNK_DB/defaultdb/colddb
151 thawedPath = $SPLUNK_DB/defaultdb/thaweddb
152 tstatsHomePath = volume:_splunk_summaries/defaultdb/datamodel_summary
153 maxMemMB = 20
154 maxConcurrentOptimizes = 6
155 maxHotIdleSecs = 86400
156 maxHotBuckets = 10
157 maxDataSize = auto_high_volume
158
159 [history]
160 homePath = $SPLUNK_DB/historydb/db
161 coldPath = $SPLUNK_DB/historydb/colddb
162 thawedPath = $SPLUNK_DB/historydb/thaweddb
163 tstatsHomePath = volume:_splunk_summaries/historydb/datamodel_summary
164 maxDataSize = 10
165 frozenTimePeriodInSecs = 604800
166
167 [summary]
168 homePath = $SPLUNK_DB/summarydb/db
169 coldPath = $SPLUNK_DB/summarydb/colddb
170 thawedPath = $SPLUNK_DB/summarydb/thaweddb
171 tstatsHomePath = volume:_splunk_summaries/summarydb/datamodel_summary
172
173 [_internal]
174 homePath = $SPLUNK_DB/_internaldb/db
175 coldPath = $SPLUNK_DB/_internaldb/colddb
176 thawedPath = $SPLUNK_DB/_internaldb/thaweddb
177 tstatsHomePath = volume:_splunk_summaries/_internaldb/datamodel_summary
178 maxDataSize = 1000
179 maxHotSpanSecs = 432000
180 frozenTimePeriodInSecs = 2592000
181
182 [_audit]
183 homePath = $SPLUNK_DB/audit/db
184 coldPath = $SPLUNK_DB/audit/colddb
185 thawedPath = $SPLUNK_DB/audit/thaweddb
186 tstatsHomePath = volume:_splunk_summaries/audit/datamodel_summary
187
188 [_thefishbucket]
189 homePath = $SPLUNK_DB/fishbucket/db
190 coldPath = $SPLUNK_DB/fishbucket/colddb
191 thawedPath = $SPLUNK_DB/fishbucket/thaweddb
192 tstatsHomePath = volume:_splunk_summaries/fishbucket/datamodel_summary
193 maxDataSize = 500
194 frozenTimePeriodInSecs = 2419200
195
196 # this index has been removed in the 4.1 series, but this stanza must be
197 # preserved to avoid displaying errors for users that have tweaked the index's
198 # size/etc parameters in local/indexes.conf.
199 #
200 [splunklogger]
201 homePath = $SPLUNK_DB/splunklogger/db
202 coldPath = $SPLUNK_DB/splunklogger/colddb
203 thawedPath = $SPLUNK_DB/splunklogger/thaweddb
204 disabled = true
205
206 [_introspection]
207 homePath = $SPLUNK_DB/_introspection/db
208 coldPath = $SPLUNK_DB/_introspection/colddb
209 thawedPath = $SPLUNK_DB/_introspection/thaweddb
210 maxDataSize = 1024
211 frozenTimePeriodInSecs = 1209600
212
213 [_telemetry]
214 homePath = $SPLUNK_DB/_telemetry/db
215 coldPath = $SPLUNK_DB/_telemetry/colddb
216 thawedPath = $SPLUNK_DB/_telemetry/thaweddb
217 maxDataSize = 256
218 frozenTimePeriodInSecs = 63072000
219
220 [_metrics]
221 homePath = $SPLUNK_DB/_metrics/db
222 coldPath = $SPLUNK_DB/_metrics/colddb
223 thawedPath = $SPLUNK_DB/_metrics/thaweddb
224 datatype = metric
225 #14 day retention
226 frozenTimePeriodInSecs = 1209600
227 splitByIndexKeys = metric_name

[aasabatini]

Hi @coweatgrass14 

Can your share the message error?

if possible you have low space on your splunk partition?

let me know

Alessandro

[coweatgrass14]

Thanks for the reply.

For disk usage:

/dev/sda1 used around 11% only

and the screen capture for the index like this:

I tried to read the error log message but there are tone of lines.... and failed to find useful error log....

[aasabatini]

Hi @coweatgrass14 

looks like disabled, can you check this conf file?

/opt/splunk/etc/system/default/indexes.conf

or 

/opt/splunk/etc/system/local/indexes.conf

check if the indexes are disabled.